2024-03-29T11:13:41.792 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T11:13:41.792 DEBUG:teuthology.run:Teuthology command: teuthology --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --verbose --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622705 --owner scheduled_amathuri@teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622705/orig.config.yaml 2024-03-29T11:13:41.853 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:13:41.944 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622705 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622705' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi135.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB3FPCTUhwsFHInF9Tv4jgUIcbdIN7FEbfb79V0LlTwdLnCP3Ohx3Nkc0l0n1RnUrjovS7OmvmSzaxp2IxMsJco= smithi194.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCGRy6yfe5LtLCQ8rJvM8g9WvDkJwb+f6+7wdIOVHmvUQjJbOpGy6Cvay+gB1EWaXZ0Us87iEr7S2+5OwPUTSLM= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T11:13:41.945 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T11:13:41.945 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T11:13:41.946 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T11:13:41.954 INFO:teuthology.task.internal:Checking packages... 2024-03-29T11:13:41.991 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T11:13:41.991 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T11:13:41.991 INFO:teuthology.packaging:ref: None 2024-03-29T11:13:41.991 INFO:teuthology.packaging:tag: None 2024-03-29T11:13:41.991 INFO:teuthology.packaging:branch: main 2024-03-29T11:13:41.991 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:13:41.992 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T11:13:42.296 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T11:13:42.298 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T11:13:42.319 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T11:13:42.319 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T11:13:42.348 INFO:teuthology.task.internal:Saving configuration 2024-03-29T11:13:42.363 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T11:13:42.401 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:13:42.428 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi135.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622705', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:08:33.968797', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB3FPCTUhwsFHInF9Tv4jgUIcbdIN7FEbfb79V0LlTwdLnCP3Ohx3Nkc0l0n1RnUrjovS7OmvmSzaxp2IxMsJco='} 2024-03-29T11:13:42.455 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622705', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:08:33.967377', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCGRy6yfe5LtLCQ8rJvM8g9WvDkJwb+f6+7wdIOVHmvUQjJbOpGy6Cvay+gB1EWaXZ0Us87iEr7S2+5OwPUTSLM='} 2024-03-29T11:13:42.455 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T11:13:42.479 INFO:teuthology.task.internal:roles: ubuntu@smithi135.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T11:13:42.479 INFO:teuthology.task.internal:roles: ubuntu@smithi194.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T11:13:42.480 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T11:13:42.610 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7f3954e0d0>, signals=[15]) 2024-03-29T11:13:42.610 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T11:13:42.632 INFO:teuthology.task.internal:Opening connections... 2024-03-29T11:13:42.632 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi135.front.sepia.ceph.com 2024-03-29T11:13:42.634 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:13:42.743 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi194.front.sepia.ceph.com 2024-03-29T11:13:42.744 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:13:42.846 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T11:13:42.880 DEBUG:teuthology.orchestra.run.smithi135:> uname -m 2024-03-29T11:13:42.908 INFO:teuthology.orchestra.run.smithi135.stdout:x86_64 2024-03-29T11:13:42.908 DEBUG:teuthology.orchestra.run.smithi135:> cat /etc/os-release 2024-03-29T11:13:42.971 INFO:teuthology.orchestra.run.smithi135.stdout:NAME="CentOS Stream" 2024-03-29T11:13:42.971 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION="9" 2024-03-29T11:13:42.971 INFO:teuthology.orchestra.run.smithi135.stdout:ID="centos" 2024-03-29T11:13:42.971 INFO:teuthology.orchestra.run.smithi135.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:13:42.971 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION_ID="9" 2024-03-29T11:13:42.971 INFO:teuthology.orchestra.run.smithi135.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:13:42.971 INFO:teuthology.orchestra.run.smithi135.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:13:42.971 INFO:teuthology.orchestra.run.smithi135.stdout:ANSI_COLOR="0;31" 2024-03-29T11:13:42.971 INFO:teuthology.orchestra.run.smithi135.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:13:42.971 INFO:teuthology.orchestra.run.smithi135.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:13:42.971 INFO:teuthology.orchestra.run.smithi135.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:13:42.971 INFO:teuthology.orchestra.run.smithi135.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:13:42.972 INFO:teuthology.orchestra.run.smithi135.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:13:42.972 INFO:teuthology.orchestra.run.smithi135.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:13:42.972 INFO:teuthology.lock.ops:Updating smithi135.front.sepia.ceph.com on lock server 2024-03-29T11:13:43.003 DEBUG:teuthology.orchestra.run.smithi194:> uname -m 2024-03-29T11:13:43.024 INFO:teuthology.orchestra.run.smithi194.stdout:x86_64 2024-03-29T11:13:43.024 DEBUG:teuthology.orchestra.run.smithi194:> cat /etc/os-release 2024-03-29T11:13:43.086 INFO:teuthology.orchestra.run.smithi194.stdout:NAME="CentOS Stream" 2024-03-29T11:13:43.086 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION="9" 2024-03-29T11:13:43.086 INFO:teuthology.orchestra.run.smithi194.stdout:ID="centos" 2024-03-29T11:13:43.086 INFO:teuthology.orchestra.run.smithi194.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:13:43.086 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_ID="9" 2024-03-29T11:13:43.087 INFO:teuthology.orchestra.run.smithi194.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:13:43.087 INFO:teuthology.orchestra.run.smithi194.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:13:43.087 INFO:teuthology.orchestra.run.smithi194.stdout:ANSI_COLOR="0;31" 2024-03-29T11:13:43.087 INFO:teuthology.orchestra.run.smithi194.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:13:43.087 INFO:teuthology.orchestra.run.smithi194.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:13:43.087 INFO:teuthology.orchestra.run.smithi194.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:13:43.087 INFO:teuthology.orchestra.run.smithi194.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:13:43.088 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:13:43.088 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:13:43.088 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server 2024-03-29T11:13:43.121 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T11:13:43.227 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T11:13:43.874 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T11:13:43.874 DEBUG:teuthology.orchestra.run.smithi135:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:13:43.881 DEBUG:teuthology.orchestra.run.smithi194:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:13:43.904 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T11:13:44.324 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T11:13:44.325 DEBUG:teuthology.orchestra.run.smithi135:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:13:44.329 DEBUG:teuthology.orchestra.run.smithi194:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:13:44.444 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T11:13:44.558 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T11:13:44.584 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T11:13:44.584 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T11:13:44.585 DEBUG:teuthology.orchestra.run.smithi135:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:13:44.585 DEBUG:teuthology.orchestra.run.smithi194:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:13:44.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:13:44.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:13:44.605 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2024-03-29T11:13:44.606 DEBUG:teuthology.orchestra.run.smithi135:> uname -r 2024-03-29T11:13:44.663 INFO:teuthology.orchestra.run.smithi194.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:13:44.663 INFO:teuthology.task.kernel:Running kernel on smithi194: 5.14.0-361.el9.x86_64 2024-03-29T11:13:44.664 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum install -y kernel 2024-03-29T11:13:44.665 INFO:teuthology.orchestra.run.smithi135.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:13:44.665 INFO:teuthology.task.kernel:Running kernel on smithi135: 5.14.0-361.el9.x86_64 2024-03-29T11:13:44.665 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum install -y kernel 2024-03-29T11:13:46.786 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-29T11:13:46.808 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-29T11:13:48.225 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-03-29T11:13:48.394 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-29T11:13:57.473 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - BaseOS 925 kB/s | 8.0 MB 00:08 2024-03-29T11:13:57.809 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - BaseOS 906 kB/s | 8.0 MB 00:09 2024-03-29T11:14:02.967 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - AppStream 4.3 MB/s | 19 MB 00:04 2024-03-29T11:14:05.284 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - AppStream 2.9 MB/s | 19 MB 00:06 2024-03-29T11:14:08.059 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.1 MB 00:01 2024-03-29T11:14:09.209 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.1 MB 00:00 2024-03-29T11:14:09.509 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - Extras packages 50 kB/s | 16 kB 00:00 2024-03-29T11:14:10.464 INFO:teuthology.orchestra.run.smithi135.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-03-29T11:14:10.860 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-03-29T11:14:12.822 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-29T11:14:15.461 INFO:teuthology.orchestra.run.smithi135.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-29T11:14:17.702 INFO:teuthology.orchestra.run.smithi135.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:14:17.702 INFO:teuthology.orchestra.run.smithi135.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:14:17.829 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-29T11:14:17.839 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:14:17.839 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-03-29T11:14:17.839 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:14:17.839 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-29T11:14:17.839 INFO:teuthology.orchestra.run.smithi135.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:14:17.839 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-03-29T11:14:17.839 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:14:17.839 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:14:17.839 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:14:17.839 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:14:17.839 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-29T11:14:17.840 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:14:17.840 INFO:teuthology.orchestra.run.smithi135.stdout:Install 4 Packages 2024-03-29T11:14:17.840 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:14:17.840 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 76 M 2024-03-29T11:14:17.840 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 123 M 2024-03-29T11:14:17.840 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-29T11:14:17.861 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 7.2 kB/s | 1.7 kB 00:00 2024-03-29T11:14:17.997 INFO:teuthology.orchestra.run.smithi135.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T11:14:18.672 INFO:teuthology.orchestra.run.smithi135.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 18 MB/s | 15 MB 00:00 2024-03-29T11:14:19.439 INFO:teuthology.orchestra.run.smithi135.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 21 MB/s | 34 MB 00:01 2024-03-29T11:14:19.856 INFO:teuthology.orchestra.run.smithi135.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 15 MB/s | 28 MB 00:01 2024-03-29T11:14:19.859 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:14:19.860 INFO:teuthology.orchestra.run.smithi135.stdout:Total 38 MB/s | 76 MB 00:02 2024-03-29T11:14:19.860 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-29T11:14:20.060 INFO:teuthology.orchestra.run.smithi194.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:14:20.060 INFO:teuthology.orchestra.run.smithi194.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:14:20.161 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-29T11:14:20.165 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:14:20.166 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-29T11:14:20.166 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:14:20.166 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-29T11:14:20.166 INFO:teuthology.orchestra.run.smithi194.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:14:20.166 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-29T11:14:20.167 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:14:20.167 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:14:20.167 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:14:20.167 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:14:20.167 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-29T11:14:20.167 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:14:20.168 INFO:teuthology.orchestra.run.smithi194.stdout:Install 4 Packages 2024-03-29T11:14:20.168 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:14:20.169 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 76 M 2024-03-29T11:14:20.169 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 123 M 2024-03-29T11:14:20.169 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-29T11:14:20.237 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-29T11:14:20.237 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-29T11:14:20.354 INFO:teuthology.orchestra.run.smithi194.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.0 MB/s | 192 kB 00:00 2024-03-29T11:14:20.823 INFO:teuthology.orchestra.run.smithi194.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 23 MB/s | 15 MB 00:00 2024-03-29T11:14:21.040 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-29T11:14:21.040 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-29T11:14:21.746 INFO:teuthology.orchestra.run.smithi194.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 21 MB/s | 34 MB 00:01 2024-03-29T11:14:22.721 INFO:teuthology.orchestra.run.smithi194.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 12 MB/s | 28 MB 00:02 2024-03-29T11:14:22.724 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:14:22.725 INFO:teuthology.orchestra.run.smithi194.stdout:Total 30 MB/s | 76 MB 00:02 2024-03-29T11:14:22.725 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-29T11:14:23.081 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-29T11:14:23.081 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-29T11:14:23.327 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-29T11:14:23.924 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-29T11:14:23.924 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-29T11:14:24.007 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:14:24.103 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:14:26.207 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-29T11:14:26.776 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:14:26.863 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:14:26.897 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:14:27.039 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:14:29.594 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:14:29.724 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:14:31.766 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:14:31.806 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:14:34.413 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:14:34.539 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:14:36.575 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:14:39.295 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:15:06.502 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:15:06.502 INFO:teuthology.orchestra.run.smithi135.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:15:06.502 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:15:06.551 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:15:06.930 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:15:06.931 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:15:06.931 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:15:06.931 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:15:07.364 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:15:07.364 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:15:07.365 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-29T11:15:07.365 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:15:07.365 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:15:07.365 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:15:07.365 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:15:07.365 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:15:07.365 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-29T11:15:07.729 DEBUG:teuthology.orchestra.run.smithi135:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:15:07.832 INFO:teuthology.orchestra.run.smithi135.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:15:07.832 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:15:07.833 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:15:07.833 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:15:07.833 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:15:07.833 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:15:07.833 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi135.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:15:07.833 DEBUG:teuthology.orchestra.run.smithi135:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:15:07.913 INFO:teuthology.orchestra.run.smithi135.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:15:07.913 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:15:07.913 DEBUG:teuthology.orchestra.run.smithi135:> sudo rpm -qi grub2-tools 2024-03-29T11:15:07.947 INFO:teuthology.orchestra.run.smithi135.stdout:Name : grub2-tools 2024-03-29T11:15:07.947 INFO:teuthology.orchestra.run.smithi135.stdout:Epoch : 1 2024-03-29T11:15:07.947 INFO:teuthology.orchestra.run.smithi135.stdout:Version : 2.06 2024-03-29T11:15:07.947 INFO:teuthology.orchestra.run.smithi135.stdout:Release : 68.el9 2024-03-29T11:15:07.948 INFO:teuthology.orchestra.run.smithi135.stdout:Architecture: x86_64 2024-03-29T11:15:07.948 INFO:teuthology.orchestra.run.smithi135.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:15:07.948 INFO:teuthology.orchestra.run.smithi135.stdout:Group : Unspecified 2024-03-29T11:15:07.948 INFO:teuthology.orchestra.run.smithi135.stdout:Size : 8264254 2024-03-29T11:15:07.948 INFO:teuthology.orchestra.run.smithi135.stdout:License : GPLv3+ 2024-03-29T11:15:07.948 INFO:teuthology.orchestra.run.smithi135.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:15:07.948 INFO:teuthology.orchestra.run.smithi135.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:15:07.948 INFO:teuthology.orchestra.run.smithi135.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:15:07.948 INFO:teuthology.orchestra.run.smithi135.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:15:07.948 INFO:teuthology.orchestra.run.smithi135.stdout:Packager : builder@centos.org 2024-03-29T11:15:07.948 INFO:teuthology.orchestra.run.smithi135.stdout:Vendor : CentOS 2024-03-29T11:15:07.948 INFO:teuthology.orchestra.run.smithi135.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:15:07.948 INFO:teuthology.orchestra.run.smithi135.stdout:Summary : Support tools for GRUB. 2024-03-29T11:15:07.948 INFO:teuthology.orchestra.run.smithi135.stdout:Description : 2024-03-29T11:15:07.949 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:15:07.949 INFO:teuthology.orchestra.run.smithi135.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:15:07.949 INFO:teuthology.orchestra.run.smithi135.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:15:07.949 INFO:teuthology.orchestra.run.smithi135.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:15:07.949 INFO:teuthology.orchestra.run.smithi135.stdout:hardware devices. 2024-03-29T11:15:07.949 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:15:07.949 INFO:teuthology.orchestra.run.smithi135.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:15:07.950 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:15:07.950 INFO:teuthology.task.kernel:Updating grub on smithi135 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:15:07.950 DEBUG:teuthology.orchestra.run.smithi135:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:15:08.281 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:15:08.282 INFO:teuthology.orchestra.run.smithi194.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:15:08.282 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:15:08.325 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:15:08.651 INFO:teuthology.orchestra.run.smithi135.stderr:Generating grub configuration file ... 2024-03-29T11:15:08.669 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:15:08.670 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:15:08.670 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:15:08.670 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:15:09.062 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:15:09.063 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:15:09.063 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-29T11:15:09.063 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:15:09.063 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:15:09.063 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:15:09.063 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:15:09.063 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:15:09.063 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-29T11:15:09.362 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:15:09.459 INFO:teuthology.orchestra.run.smithi194.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:15:09.459 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:15:09.459 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:15:09.459 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:15:09.459 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:15:09.459 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:15:09.459 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi194.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:15:09.459 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:15:09.539 INFO:teuthology.orchestra.run.smithi194.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:15:09.540 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:15:09.540 DEBUG:teuthology.orchestra.run.smithi194:> sudo rpm -qi grub2-tools 2024-03-29T11:15:09.574 INFO:teuthology.orchestra.run.smithi194.stdout:Name : grub2-tools 2024-03-29T11:15:09.574 INFO:teuthology.orchestra.run.smithi194.stdout:Epoch : 1 2024-03-29T11:15:09.574 INFO:teuthology.orchestra.run.smithi194.stdout:Version : 2.06 2024-03-29T11:15:09.574 INFO:teuthology.orchestra.run.smithi194.stdout:Release : 68.el9 2024-03-29T11:15:09.574 INFO:teuthology.orchestra.run.smithi194.stdout:Architecture: x86_64 2024-03-29T11:15:09.574 INFO:teuthology.orchestra.run.smithi194.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:15:09.574 INFO:teuthology.orchestra.run.smithi194.stdout:Group : Unspecified 2024-03-29T11:15:09.574 INFO:teuthology.orchestra.run.smithi194.stdout:Size : 8264254 2024-03-29T11:15:09.574 INFO:teuthology.orchestra.run.smithi194.stdout:License : GPLv3+ 2024-03-29T11:15:09.575 INFO:teuthology.orchestra.run.smithi194.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:15:09.575 INFO:teuthology.orchestra.run.smithi194.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:15:09.575 INFO:teuthology.orchestra.run.smithi194.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:15:09.575 INFO:teuthology.orchestra.run.smithi194.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:15:09.575 INFO:teuthology.orchestra.run.smithi194.stdout:Packager : builder@centos.org 2024-03-29T11:15:09.575 INFO:teuthology.orchestra.run.smithi194.stdout:Vendor : CentOS 2024-03-29T11:15:09.575 INFO:teuthology.orchestra.run.smithi194.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:15:09.575 INFO:teuthology.orchestra.run.smithi194.stdout:Summary : Support tools for GRUB. 2024-03-29T11:15:09.575 INFO:teuthology.orchestra.run.smithi194.stdout:Description : 2024-03-29T11:15:09.575 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:15:09.575 INFO:teuthology.orchestra.run.smithi194.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:15:09.575 INFO:teuthology.orchestra.run.smithi194.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:15:09.575 INFO:teuthology.orchestra.run.smithi194.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:15:09.575 INFO:teuthology.orchestra.run.smithi194.stdout:hardware devices. 2024-03-29T11:15:09.576 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:15:09.576 INFO:teuthology.orchestra.run.smithi194.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:15:09.576 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:15:09.577 INFO:teuthology.task.kernel:Updating grub on smithi194 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:15:09.577 DEBUG:teuthology.orchestra.run.smithi194:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:15:10.031 INFO:teuthology.orchestra.run.smithi135.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:15:10.066 INFO:teuthology.orchestra.run.smithi135.stderr:done 2024-03-29T11:15:10.069 DEBUG:teuthology.orchestra.run.smithi135:> mktemp 2024-03-29T11:15:10.084 INFO:teuthology.orchestra.run.smithi135.stdout:/tmp/tmp.HQpOhCX3Oc 2024-03-29T11:15:10.084 DEBUG:teuthology.orchestra.run.smithi135:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.HQpOhCX3Oc 2024-03-29T11:15:10.153 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 0666 /tmp/tmp.HQpOhCX3Oc 2024-03-29T11:15:10.179 INFO:teuthology.orchestra.run.smithi194.stderr:Generating grub configuration file ... 2024-03-29T11:15:10.295 DEBUG:teuthology.orchestra.remote:smithi135:/tmp/tmp.HQpOhCX3Oc is 6KB 2024-03-29T11:15:10.344 DEBUG:teuthology.orchestra.run.smithi135:> rm -fr /tmp/tmp.HQpOhCX3Oc 2024-03-29T11:15:10.359 DEBUG:teuthology.orchestra.run.smithi135:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:15:10.427 INFO:teuthology.orchestra.run.smithi135.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:15:10.427 INFO:teuthology.orchestra.run.smithi135.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:15:10.427 INFO:teuthology.orchestra.run.smithi135.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:15:10.427 INFO:teuthology.orchestra.run.smithi135.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:15:10.428 DEBUG:teuthology.orchestra.run.smithi135:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:15:10.588 DEBUG:teuthology.orchestra.run.smithi135:> sudo shutdown -r now 2024-03-29T11:15:11.582 INFO:teuthology.orchestra.run.smithi194.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:15:11.621 INFO:teuthology.orchestra.run.smithi194.stderr:done 2024-03-29T11:15:11.623 DEBUG:teuthology.orchestra.run.smithi194:> mktemp 2024-03-29T11:15:11.638 INFO:teuthology.orchestra.run.smithi194.stdout:/tmp/tmp.dCA09QwYJR 2024-03-29T11:15:11.638 DEBUG:teuthology.orchestra.run.smithi194:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.dCA09QwYJR 2024-03-29T11:15:11.707 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0666 /tmp/tmp.dCA09QwYJR 2024-03-29T11:15:11.850 DEBUG:teuthology.orchestra.remote:smithi194:/tmp/tmp.dCA09QwYJR is 6KB 2024-03-29T11:15:11.898 DEBUG:teuthology.orchestra.run.smithi194:> rm -fr /tmp/tmp.dCA09QwYJR 2024-03-29T11:15:11.913 DEBUG:teuthology.orchestra.run.smithi194:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:15:11.977 INFO:teuthology.orchestra.run.smithi194.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:15:11.977 INFO:teuthology.orchestra.run.smithi194.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:15:11.977 INFO:teuthology.orchestra.run.smithi194.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:15:11.977 INFO:teuthology.orchestra.run.smithi194.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:15:11.979 DEBUG:teuthology.orchestra.run.smithi194:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:15:12.119 DEBUG:teuthology.orchestra.run.smithi194:> sudo shutdown -r now 2024-03-29T11:15:40.618 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:15:40.619 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-03-29T11:15:40.620 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:15:42.122 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:15:42.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-03-29T11:15:42.123 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:16:15.263 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.135 2024-03-29T11:16:24.272 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-03-29T11:16:24.273 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:16:42.145 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:16:42.654 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.135 2024-03-29T11:16:51.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-03-29T11:16:51.152 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:16:54.602 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-03-29T11:16:54.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-03-29T11:16:54.656 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:16:55.143 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-03-29T11:16:55.144 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:16:55.144 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2024-03-29T11:16:55.158 INFO:teuthology.orchestra.run.smithi194.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:16:55.158 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:16:55.159 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:16:55.159 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:16:55.303 DEBUG:teuthology.orchestra.run.smithi135:> true 2024-03-29T11:16:55.788 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-03-29T11:16:55.788 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:16:55.788 DEBUG:teuthology.orchestra.run.smithi135:> uname -r 2024-03-29T11:16:55.802 INFO:teuthology.orchestra.run.smithi135.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:16:55.803 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:16:55.803 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:16:55.803 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:16:56.159 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:16:56.160 DEBUG:teuthology.orchestra.run.smithi194:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:16:56.236 INFO:teuthology.orchestra.run.smithi194.stdout:ttyS1 2024-03-29T11:16:56.274 DEBUG:teuthology.parallel:result is None 2024-03-29T11:16:56.804 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:16:56.804 DEBUG:teuthology.orchestra.run.smithi135:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:16:56.899 INFO:teuthology.orchestra.run.smithi135.stdout:ttyS1 2024-03-29T11:16:56.937 DEBUG:teuthology.parallel:result is None 2024-03-29T11:16:56.938 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:16:56.945 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:16:56.946 DEBUG:teuthology.orchestra.run.smithi135:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:16:56.948 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:16:56.963 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:16:56.970 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:16:56.982 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:16:56.983 DEBUG:teuthology.orchestra.run.smithi135:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:16:57.004 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:16:57.037 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:16:57.044 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:16:57.045 DEBUG:teuthology.orchestra.run.smithi135:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:16:57.067 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:16:57.101 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:16:57.104 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:16:57.114 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:16:57.117 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:16:57.119 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:16:57.126 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:16:57.126 DEBUG:teuthology.orchestra.run.smithi135:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:16:57.161 DEBUG:teuthology.orchestra.run.smithi194:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:16:57.189 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:16:57.199 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:16:57.199 DEBUG:teuthology.orchestra.run.smithi135:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:16:57.229 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:16:57.244 DEBUG:teuthology.orchestra.run.smithi135:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:16:57.326 DEBUG:teuthology.orchestra.run.smithi135:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:16:57.372 DEBUG:teuthology.orchestra.run.smithi135:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:16:57.453 DEBUG:teuthology.orchestra.run.smithi135:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:16:57.497 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-29T11:16:57.497 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:16:57.565 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:16:57.608 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:16:57.654 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:16:57.736 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:16:57.778 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-29T11:16:57.778 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:16:57.846 DEBUG:teuthology.orchestra.run.smithi135:> sudo service rsyslog restart 2024-03-29T11:16:57.848 DEBUG:teuthology.orchestra.run.smithi194:> sudo service rsyslog restart 2024-03-29T11:16:57.899 INFO:teuthology.orchestra.run.smithi135.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:16:57.934 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:16:58.215 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:16:58.222 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:16:58.222 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:16:58.234 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:16:58.241 DEBUG:teuthology.orchestra.run.smithi135:> sudo service auditd rotate 2024-03-29T11:16:58.327 INFO:teuthology.orchestra.run.smithi135.stdout:Rotating logs: 2024-03-29T11:16:58.329 DEBUG:teuthology.orchestra.run.smithi194:> sudo service auditd rotate 2024-03-29T11:16:58.387 INFO:teuthology.orchestra.run.smithi194.stdout:Rotating logs: 2024-03-29T11:16:58.389 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:16:58.389 DEBUG:teuthology.orchestra.run.smithi135:> /usr/sbin/getenforce 2024-03-29T11:16:58.404 INFO:teuthology.orchestra.run.smithi135.stdout:Permissive 2024-03-29T11:16:58.404 DEBUG:teuthology.orchestra.run.smithi194:> /usr/sbin/getenforce 2024-03-29T11:16:58.448 INFO:teuthology.orchestra.run.smithi194.stdout:Permissive 2024-03-29T11:16:58.448 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi135.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi194.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:16:58.449 DEBUG:teuthology.orchestra.run.smithi135:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:16:58.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:16:58.476 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:16:58.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:16:58.519 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:16:58.519 DEBUG:teuthology.orchestra.run.smithi135:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:16:58.557 DEBUG:teuthology.orchestra.run.smithi194:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:16:58.598 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:16:58.608 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:16:58.624 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:16:58.625 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi135.front.sepia.ceph.com,smithi194.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:21:09.505 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi135.front.sepia.ceph.com'), Remote(name='ubuntu@smithi194.front.sepia.ceph.com')] 2024-03-29T11:21:09.507 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-03-29T11:21:09.508 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:21:09.596 DEBUG:teuthology.orchestra.run.smithi135:> true 2024-03-29T11:21:09.672 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi135.front.sepia.ceph.com' 2024-03-29T11:21:09.673 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-03-29T11:21:09.673 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:21:09.763 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-03-29T11:21:09.836 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-03-29T11:21:09.837 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:21:09.847 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:21:09.847 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:21:09.847 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:21:09.850 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:21:09.850 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:21:09.878 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:21:09.894 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:21:09.907 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:21:09.924 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: ntpd: command not found 2024-03-29T11:21:09.925 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:21:09.938 INFO:teuthology.orchestra.run.smithi135.stdout:506 Cannot talk to daemon 2024-03-29T11:21:09.953 INFO:teuthology.orchestra.run.smithi194.stderr:sudo: ntpd: command not found 2024-03-29T11:21:09.954 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:21:09.967 INFO:teuthology.orchestra.run.smithi194.stdout:506 Cannot talk to daemon 2024-03-29T11:21:09.971 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:21:09.983 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:21:10.000 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:21:10.024 INFO:teuthology.orchestra.run.smithi135.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:21:10.027 INFO:teuthology.orchestra.run.smithi135.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:21:10.027 INFO:teuthology.orchestra.run.smithi135.stdout:=============================================================================== 2024-03-29T11:21:10.027 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:21:10.028 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:21:10.028 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:21:10.028 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:21:10.053 INFO:teuthology.orchestra.run.smithi194.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:21:10.056 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:21:10.057 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-03-29T11:21:10.057 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:21:10.057 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:21:10.057 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:21:10.057 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:21:10.057 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:21:10.068 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:21:10.069 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:21:10.069 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:21:10.069 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:21:10.074 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:21:10.075 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:21:10.075 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:21:10.076 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:21:10.078 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:21:10.078 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:21:10.253 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:21:10.253 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:21:10.323 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:21:10.323 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:21:10.446 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:21:10.447 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-29T11:21:10.447 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:21:10.453 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:21:10.453 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-29T11:21:10.453 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:21:10.475 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:21:10.475 DEBUG:teuthology.orchestra.run.smithi194:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:21:10.481 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:21:10.481 DEBUG:teuthology.orchestra.run.smithi135:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:21:10.543 DEBUG:teuthology.orchestra.run.smithi194:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:21:10.549 DEBUG:teuthology.orchestra.run.smithi135:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:21:10.622 DEBUG:teuthology.orchestra.run.smithi194:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:21:10.628 DEBUG:teuthology.orchestra.run.smithi135:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:21:10.689 INFO:teuthology.orchestra.run.smithi194.stdout:check_obsoletes = 1 2024-03-29T11:21:10.690 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum clean all 2024-03-29T11:21:10.693 INFO:teuthology.orchestra.run.smithi135.stdout:check_obsoletes = 1 2024-03-29T11:21:10.695 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum clean all 2024-03-29T11:21:10.930 INFO:teuthology.orchestra.run.smithi194.stdout:74 files removed 2024-03-29T11:21:10.943 INFO:teuthology.orchestra.run.smithi135.stdout:74 files removed 2024-03-29T11:21:10.956 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-radosgw 2024-03-29T11:21:10.973 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-radosgw 2024-03-29T11:21:11.758 INFO:teuthology.orchestra.run.smithi135.stdout:ceph packages for x86_64 169 kB/s | 82 kB 00:00 2024-03-29T11:21:11.837 INFO:teuthology.orchestra.run.smithi194.stdout:ceph packages for x86_64 141 kB/s | 82 kB 00:00 2024-03-29T11:21:12.296 INFO:teuthology.orchestra.run.smithi194.stdout:ceph noarch packages 24 kB/s | 10 kB 00:00 2024-03-29T11:21:12.334 INFO:teuthology.orchestra.run.smithi135.stdout:ceph noarch packages 19 kB/s | 10 kB 00:00 2024-03-29T11:21:12.850 INFO:teuthology.orchestra.run.smithi135.stdout:ceph source packages 4.0 kB/s | 1.9 kB 00:00 2024-03-29T11:21:12.930 INFO:teuthology.orchestra.run.smithi194.stdout:ceph source packages 3.2 kB/s | 1.9 kB 00:00 2024-03-29T11:21:13.936 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS-9 - AppStream 6.4 MB/s | 6.7 MB 00:01 2024-03-29T11:21:14.074 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS-9 - AppStream 6.0 MB/s | 6.7 MB 00:01 2024-03-29T11:21:15.737 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS-9 - BaseOS 3.4 MB/s | 2.0 MB 00:00 2024-03-29T11:21:15.866 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS-9 - BaseOS 3.0 MB/s | 2.0 MB 00:00 2024-03-29T11:21:16.840 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-29T11:21:16.852 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-29T11:21:19.912 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - AppStream 9.6 MB/s | 19 MB 00:01 2024-03-29T11:21:20.916 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - AppStream 6.3 MB/s | 19 MB 00:03 2024-03-29T11:21:25.599 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - CRB 5.2 MB/s | 6.1 MB 00:01 2024-03-29T11:21:27.142 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - Extras packages 84 kB/s | 16 kB 00:00 2024-03-29T11:21:28.479 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-29T11:21:31.453 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - CRB 780 kB/s | 6.1 MB 00:08 2024-03-29T11:21:33.095 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-03-29T11:21:33.626 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 4.9 kB/s | 1.7 kB 00:00 2024-03-29T11:21:34.491 INFO:teuthology.orchestra.run.smithi135.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-29T11:21:36.229 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-29T11:21:36.241 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:21:36.241 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-29T11:21:36.241 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:21:36.241 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-29T11:21:36.241 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:21:36.241 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrading: 2024-03-29T11:21:36.241 INFO:teuthology.orchestra.run.smithi194.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:21:36.242 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:21:36.242 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-29T11:21:36.242 INFO:teuthology.orchestra.run.smithi194.stdout: boost-program-options 2024-03-29T11:21:36.242 INFO:teuthology.orchestra.run.smithi194.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:21:36.242 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:21:36.242 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:21:36.242 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:21:36.242 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:21:36.242 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:21:36.242 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:21:36.242 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:21:36.242 INFO:teuthology.orchestra.run.smithi194.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:21:36.243 INFO:teuthology.orchestra.run.smithi194.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:21:36.243 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:21:36.243 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:21:36.243 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:21:36.243 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:21:36.243 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:21:36.243 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:21:36.243 INFO:teuthology.orchestra.run.smithi194.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:21:36.243 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse 2024-03-29T11:21:36.243 INFO:teuthology.orchestra.run.smithi194.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:21:36.243 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:21:36.243 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:21:36.243 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:21:36.244 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:21:36.244 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:21:36.244 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:21:36.244 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:21:36.244 INFO:teuthology.orchestra.run.smithi194.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:21:36.244 INFO:teuthology.orchestra.run.smithi194.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:21:36.244 INFO:teuthology.orchestra.run.smithi194.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:21:36.244 INFO:teuthology.orchestra.run.smithi194.stdout:Installing weak dependencies: 2024-03-29T11:21:36.244 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:21:36.244 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:21:36.244 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-29T11:21:36.244 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:21:36.244 INFO:teuthology.orchestra.run.smithi194.stdout:Install 30 Packages 2024-03-29T11:21:36.245 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrade 2 Packages 2024-03-29T11:21:36.245 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:21:36.245 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 58 M 2024-03-29T11:21:36.245 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-29T11:21:36.819 INFO:teuthology.orchestra.run.smithi194.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 10 MB/s | 5.3 MB 00:00 2024-03-29T11:21:36.861 INFO:teuthology.orchestra.run.smithi194.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 607 kB/s | 25 kB 00:00 2024-03-29T11:21:36.936 INFO:teuthology.orchestra.run.smithi194.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 9.7 MB/s | 742 kB 00:00 2024-03-29T11:21:37.003 INFO:teuthology.orchestra.run.smithi194.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 7.3 MB/s | 498 kB 00:00 2024-03-29T11:21:37.328 INFO:teuthology.orchestra.run.smithi194.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 16 MB/s | 5.1 MB 00:00 2024-03-29T11:21:37.379 INFO:teuthology.orchestra.run.smithi194.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 905 kB/s | 45 kB 00:00 2024-03-29T11:21:37.437 INFO:teuthology.orchestra.run.smithi194.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 2.2 MB/s | 131 kB 00:00 2024-03-29T11:21:37.496 INFO:teuthology.orchestra.run.smithi194.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.7 MB/s | 162 kB 00:00 2024-03-29T11:21:37.579 INFO:teuthology.orchestra.run.smithi194.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 3.8 MB/s | 320 kB 00:00 2024-03-29T11:21:37.646 INFO:teuthology.orchestra.run.smithi194.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 4.4 MB/s | 298 kB 00:00 2024-03-29T11:21:37.705 INFO:teuthology.orchestra.run.smithi194.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.7 MB/s | 99 kB 00:00 2024-03-29T11:21:37.856 INFO:teuthology.orchestra.run.smithi194.stdout:(12/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 6.2 MB/s | 9.7 MB 00:01 2024-03-29T11:21:37.950 INFO:teuthology.orchestra.run.smithi194.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 432 kB/s | 106 kB 00:00 2024-03-29T11:21:37.990 INFO:teuthology.orchestra.run.smithi194.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2024-03-29T11:21:38.014 INFO:teuthology.orchestra.run.smithi194.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 695 kB/s | 44 kB 00:00 2024-03-29T11:21:38.164 INFO:teuthology.orchestra.run.smithi194.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 661 kB 00:00 2024-03-29T11:21:38.206 INFO:teuthology.orchestra.run.smithi194.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 282 kB 00:00 2024-03-29T11:21:38.231 INFO:teuthology.orchestra.run.smithi194.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 621 kB/s | 41 kB 00:00 2024-03-29T11:21:38.265 INFO:teuthology.orchestra.run.smithi194.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 220 kB 00:00 2024-03-29T11:21:38.290 INFO:teuthology.orchestra.run.smithi194.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 596 kB/s | 35 kB 00:00 2024-03-29T11:21:38.316 INFO:teuthology.orchestra.run.smithi194.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 925 kB/s | 46 kB 00:00 2024-03-29T11:21:38.758 INFO:teuthology.orchestra.run.smithi194.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 662 kB/s | 309 kB 00:00 2024-03-29T11:21:38.849 INFO:teuthology.orchestra.run.smithi194.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 284 kB/s | 26 kB 00:00 2024-03-29T11:21:38.933 INFO:teuthology.orchestra.run.smithi194.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 582 kB/s | 49 kB 00:00 2024-03-29T11:21:39.011 INFO:teuthology.orchestra.run.smithi194.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 6.4 MB/s | 4.4 MB 00:00 2024-03-29T11:21:39.034 INFO:teuthology.orchestra.run.smithi194.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 672 kB/s | 67 kB 00:00 2024-03-29T11:21:39.092 INFO:teuthology.orchestra.run.smithi194.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-03-29T11:21:39.126 INFO:teuthology.orchestra.run.smithi194.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-03-29T11:21:39.193 INFO:teuthology.orchestra.run.smithi194.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-29T11:21:39.489 INFO:teuthology.orchestra.run.smithi135.stdout:lab-extras 44 kB/s | 1.7 kB 00:00 2024-03-29T11:21:39.510 INFO:teuthology.orchestra.run.smithi194.stdout:(30/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 6.2 MB/s | 20 MB 00:03 2024-03-29T11:21:39.685 INFO:teuthology.orchestra.run.smithi194.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 6.0 MB/s | 3.4 MB 00:00 2024-03-29T11:21:39.877 INFO:teuthology.orchestra.run.smithi194.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 4.6 MB/s | 3.1 MB 00:00 2024-03-29T11:21:39.881 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:21:39.882 INFO:teuthology.orchestra.run.smithi194.stdout:Total 16 MB/s | 58 MB 00:03 2024-03-29T11:21:39.882 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-29T11:21:39.919 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-29T11:21:39.919 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-29T11:21:40.394 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-29T11:21:40.394 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-29T11:21:41.859 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-29T11:21:41.942 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:21:41.983 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:21:42.007 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:21:42.038 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:21:42.054 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:21:42.094 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-29T11:21:42.104 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:21:42.104 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-03-29T11:21:42.104 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:21:42.104 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-29T11:21:42.104 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:21:42.104 INFO:teuthology.orchestra.run.smithi135.stdout:Upgrading: 2024-03-29T11:21:42.105 INFO:teuthology.orchestra.run.smithi135.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:21:42.105 INFO:teuthology.orchestra.run.smithi135.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:21:42.105 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-03-29T11:21:42.105 INFO:teuthology.orchestra.run.smithi135.stdout: boost-program-options 2024-03-29T11:21:42.105 INFO:teuthology.orchestra.run.smithi135.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:21:42.105 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:21:42.105 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:21:42.105 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:21:42.105 INFO:teuthology.orchestra.run.smithi135.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:21:42.105 INFO:teuthology.orchestra.run.smithi135.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:21:42.105 INFO:teuthology.orchestra.run.smithi135.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:21:42.105 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:21:42.105 INFO:teuthology.orchestra.run.smithi135.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:21:42.105 INFO:teuthology.orchestra.run.smithi135.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:21:42.106 INFO:teuthology.orchestra.run.smithi135.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:21:42.106 INFO:teuthology.orchestra.run.smithi135.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:21:42.106 INFO:teuthology.orchestra.run.smithi135.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:21:42.106 INFO:teuthology.orchestra.run.smithi135.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:21:42.106 INFO:teuthology.orchestra.run.smithi135.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:21:42.106 INFO:teuthology.orchestra.run.smithi135.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:21:42.106 INFO:teuthology.orchestra.run.smithi135.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:21:42.106 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-argparse 2024-03-29T11:21:42.106 INFO:teuthology.orchestra.run.smithi135.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:21:42.106 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:21:42.106 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:21:42.106 INFO:teuthology.orchestra.run.smithi135.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:21:42.106 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:21:42.107 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:21:42.107 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:21:42.107 INFO:teuthology.orchestra.run.smithi135.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:21:42.107 INFO:teuthology.orchestra.run.smithi135.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:21:42.107 INFO:teuthology.orchestra.run.smithi135.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:21:42.107 INFO:teuthology.orchestra.run.smithi135.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:21:42.107 INFO:teuthology.orchestra.run.smithi135.stdout:Installing weak dependencies: 2024-03-29T11:21:42.107 INFO:teuthology.orchestra.run.smithi135.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:21:42.107 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:21:42.107 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-29T11:21:42.107 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:21:42.107 INFO:teuthology.orchestra.run.smithi135.stdout:Install 30 Packages 2024-03-29T11:21:42.107 INFO:teuthology.orchestra.run.smithi135.stdout:Upgrade 2 Packages 2024-03-29T11:21:42.108 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:21:42.108 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 58 M 2024-03-29T11:21:42.108 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-29T11:21:42.248 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:21:42.310 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:21:42.638 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:21:42.709 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:21:42.801 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:21:42.889 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:21:42.910 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:21:42.956 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:21:42.985 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:21:43.020 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:21:43.090 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:21:43.121 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:21:43.176 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:21:43.211 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:21:43.318 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:21:43.353 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:21:43.394 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:21:43.426 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:21:43.455 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:21:43.498 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:21:43.825 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:21:43.859 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:21:43.932 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:21:43.964 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:21:44.005 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:21:44.248 INFO:teuthology.orchestra.run.smithi135.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 4.7 MB/s | 9.7 MB 00:02 2024-03-29T11:21:44.315 INFO:teuthology.orchestra.run.smithi135.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 379 kB/s | 25 kB 00:00 2024-03-29T11:21:44.467 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:21:44.495 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:21:44.523 INFO:teuthology.orchestra.run.smithi135.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 3.5 MB/s | 742 kB 00:00 2024-03-29T11:21:44.566 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:21:44.635 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:21:44.664 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:21:44.690 INFO:teuthology.orchestra.run.smithi135.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 2.9 MB/s | 498 kB 00:00 2024-03-29T11:21:45.174 INFO:teuthology.orchestra.run.smithi135.stdout:(5/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 1.8 MB/s | 5.3 MB 00:03 2024-03-29T11:21:45.299 INFO:teuthology.orchestra.run.smithi135.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 361 kB/s | 45 kB 00:00 2024-03-29T11:21:45.391 INFO:teuthology.orchestra.run.smithi135.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.4 MB/s | 131 kB 00:00 2024-03-29T11:21:45.541 INFO:teuthology.orchestra.run.smithi135.stdout:(8/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 6.0 MB/s | 5.1 MB 00:00 2024-03-29T11:21:45.758 INFO:teuthology.orchestra.run.smithi135.stdout:(9/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 5.5 MB/s | 20 MB 00:03 2024-03-29T11:21:45.783 INFO:teuthology.orchestra.run.smithi135.stdout:(10/32): python3-cephfs-19.0.0-1439.g5dcb190d.e 412 kB/s | 162 kB 00:00 2024-03-29T11:21:45.808 INFO:teuthology.orchestra.run.smithi135.stdout:(11/32): python3-rados-19.0.0-1439.g5dcb190d.el 1.2 MB/s | 320 kB 00:00 2024-03-29T11:21:45.834 INFO:teuthology.orchestra.run.smithi135.stdout:(12/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 3.9 MB/s | 298 kB 00:00 2024-03-29T11:21:45.917 INFO:teuthology.orchestra.run.smithi135.stdout:(13/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 743 kB/s | 99 kB 00:00 2024-03-29T11:21:45.959 INFO:teuthology.orchestra.run.smithi135.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 702 kB/s | 106 kB 00:00 2024-03-29T11:21:45.985 INFO:teuthology.orchestra.run.smithi135.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-29T11:21:46.001 INFO:teuthology.orchestra.run.smithi135.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 534 kB/s | 44 kB 00:00 2024-03-29T11:21:46.093 INFO:teuthology.orchestra.run.smithi135.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 661 kB 00:00 2024-03-29T11:21:46.118 INFO:teuthology.orchestra.run.smithi135.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 282 kB 00:00 2024-03-29T11:21:46.135 INFO:teuthology.orchestra.run.smithi135.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 309 kB/s | 41 kB 00:00 2024-03-29T11:21:46.169 INFO:teuthology.orchestra.run.smithi135.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.8 MB/s | 220 kB 00:00 2024-03-29T11:21:46.186 INFO:teuthology.orchestra.run.smithi135.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 922 kB/s | 46 kB 00:00 2024-03-29T11:21:46.203 INFO:teuthology.orchestra.run.smithi135.stdout:(22/32): qatlib-service-24.02.0-1.el9.x86_64.rp 518 kB/s | 35 kB 00:00 2024-03-29T11:21:46.378 INFO:teuthology.orchestra.run.smithi135.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 206 kB/s | 26 kB 00:00 2024-03-29T11:21:46.404 INFO:teuthology.orchestra.run.smithi135.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.3 MB/s | 309 kB 00:00 2024-03-29T11:21:46.437 INFO:teuthology.orchestra.run.smithi135.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 834 kB/s | 49 kB 00:00 2024-03-29T11:21:46.462 INFO:teuthology.orchestra.run.smithi135.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-29T11:21:46.630 INFO:teuthology.orchestra.run.smithi135.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.3 MB/s | 838 kB 00:00 2024-03-29T11:21:46.663 INFO:teuthology.orchestra.run.smithi135.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 960 kB/s | 192 kB 00:00 2024-03-29T11:21:46.806 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:21:46.830 INFO:teuthology.orchestra.run.smithi135.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 6.9 MB/s | 4.4 MB 00:00 2024-03-29T11:21:46.852 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:21:46.872 INFO:teuthology.orchestra.run.smithi135.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 6.6 MB/s | 1.6 MB 00:00 2024-03-29T11:21:47.122 INFO:teuthology.orchestra.run.smithi135.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 11 MB/s | 3.1 MB 00:00 2024-03-29T11:21:47.399 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:21:47.439 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:21:47.532 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:21:47.532 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:21:47.532 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:21:47.533 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:21:47.565 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:21:52.105 INFO:teuthology.orchestra.run.smithi135.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 635 kB/s | 3.4 MB 00:05 2024-03-29T11:21:52.113 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:21:52.113 INFO:teuthology.orchestra.run.smithi135.stdout:Total 5.8 MB/s | 58 MB 00:10 2024-03-29T11:21:52.113 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-29T11:21:52.150 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-29T11:21:52.151 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-29T11:21:52.629 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-29T11:21:52.629 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-29T11:21:54.078 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-29T11:21:54.166 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:21:54.202 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:21:54.227 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:21:54.250 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:21:54.266 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:21:54.467 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:21:54.530 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:21:54.849 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:21:54.895 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:21:54.987 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:21:55.075 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:21:55.097 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:21:55.142 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:21:55.172 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:21:55.216 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:21:55.301 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:21:55.332 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:21:55.396 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:21:55.439 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:21:55.549 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:21:55.599 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:21:55.639 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:21:55.672 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:21:55.701 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:21:55.733 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:21:56.054 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:21:56.088 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:21:56.195 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:21:56.226 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:21:56.275 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:21:56.733 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:21:56.766 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:21:56.820 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:21:56.888 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:21:56.918 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:21:57.154 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:21:57.155 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /sys 2024-03-29T11:21:57.155 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /proc 2024-03-29T11:21:57.155 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /mnt 2024-03-29T11:21:57.155 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /var/tmp 2024-03-29T11:21:57.155 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /home 2024-03-29T11:21:57.155 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /root 2024-03-29T11:21:57.155 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /tmp 2024-03-29T11:21:57.155 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:21:58.056 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:21:58.082 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:21:58.082 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:21:58.082 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:21:58.082 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:21:58.082 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:21:58.082 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:21:58.083 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:21:58.167 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:21:58.167 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:21:58.991 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:21:59.039 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:21:59.316 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:21:59.316 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:21:59.316 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:21:59.316 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:21:59.316 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:21:59.316 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:21:59.317 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:21:59.317 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:21:59.317 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:21:59.317 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:21:59.317 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:21:59.317 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:21:59.317 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:21:59.317 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:21:59.317 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:21:59.317 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:21:59.317 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:21:59.317 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:21:59.318 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:21:59.318 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:21:59.318 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:21:59.318 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:21:59.318 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:21:59.318 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:21:59.318 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:21:59.318 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:21:59.318 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:21:59.318 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:21:59.318 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:21:59.318 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:21:59.318 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:21:59.319 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:21:59.319 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:21:59.319 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:21:59.602 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:21:59.642 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:21:59.736 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:21:59.737 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:21:59.737 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:21:59.737 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:21:59.777 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:22:00.518 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:22:00.518 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:00.518 INFO:teuthology.orchestra.run.smithi194.stdout:Upgraded: 2024-03-29T11:22:00.518 INFO:teuthology.orchestra.run.smithi194.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:00.518 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:00.518 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-29T11:22:00.519 INFO:teuthology.orchestra.run.smithi194.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:22:00.519 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:00.519 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:00.519 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:00.519 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:00.519 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:22:00.519 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:22:00.519 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:22:00.519 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:00.519 INFO:teuthology.orchestra.run.smithi194.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:22:00.519 INFO:teuthology.orchestra.run.smithi194.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:22:00.519 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:22:00.519 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:00.520 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:22:00.520 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:00.520 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:22:00.520 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:22:00.520 INFO:teuthology.orchestra.run.smithi194.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:22:00.520 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:00.520 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:00.520 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:00.520 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:22:00.520 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:00.520 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:00.521 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:00.521 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:22:00.521 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:22:00.521 INFO:teuthology.orchestra.run.smithi194.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:22:00.521 INFO:teuthology.orchestra.run.smithi194.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:22:00.521 INFO:teuthology.orchestra.run.smithi194.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:22:00.521 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:00.521 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-29T11:22:00.809 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-test 2024-03-29T11:22:01.365 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:28 ago on Fri 29 Mar 2024 11:21:33 AM UTC. 2024-03-29T11:22:01.514 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-29T11:22:01.515 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:22:01.515 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-29T11:22:01.516 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:22:01.516 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-29T11:22:01.516 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:22:01.516 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-29T11:22:01.516 INFO:teuthology.orchestra.run.smithi194.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:22:01.516 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:22:01.516 INFO:teuthology.orchestra.run.smithi194.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:22:01.516 INFO:teuthology.orchestra.run.smithi194.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:22:01.516 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:22:01.516 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:01.516 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-29T11:22:01.516 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:22:01.516 INFO:teuthology.orchestra.run.smithi194.stdout:Install 6 Packages 2024-03-29T11:22:01.516 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:01.517 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 48 M 2024-03-29T11:22:01.517 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 204 M 2024-03-29T11:22:01.517 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-29T11:22:02.087 INFO:teuthology.orchestra.run.smithi194.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 713 kB/s | 217 kB 00:00 2024-03-29T11:22:02.296 INFO:teuthology.orchestra.run.smithi194.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-03-29T11:22:02.371 INFO:teuthology.orchestra.run.smithi194.stdout:(3/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 277 kB/s | 163 kB 00:00 2024-03-29T11:22:02.396 INFO:teuthology.orchestra.run.smithi194.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 624 kB/s | 63 kB 00:00 2024-03-29T11:22:02.521 INFO:teuthology.orchestra.run.smithi194.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-29T11:22:09.516 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:22:09.516 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /sys 2024-03-29T11:22:09.516 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /proc 2024-03-29T11:22:09.516 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /mnt 2024-03-29T11:22:09.516 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /var/tmp 2024-03-29T11:22:09.516 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /home 2024-03-29T11:22:09.516 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /root 2024-03-29T11:22:09.516 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /tmp 2024-03-29T11:22:09.516 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:10.435 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:22:10.460 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:22:10.460 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:22:10.460 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:22:10.460 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:22:10.460 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:22:10.460 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:10.461 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:22:10.554 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:22:10.554 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:22:11.727 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:22:11.727 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:22:11.727 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:22:11.727 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:22:11.727 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:22:11.728 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:22:11.728 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:22:11.728 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:22:11.728 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:22:11.728 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:22:11.729 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:22:11.729 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:22:11.729 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:22:11.729 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:22:11.729 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:22:11.729 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:22:11.730 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:22:11.730 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:22:11.730 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:22:11.730 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:22:11.730 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:22:11.731 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:22:11.731 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:22:11.731 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:22:11.731 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:22:11.731 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:22:11.731 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:22:11.732 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:22:11.732 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:22:11.732 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:22:11.732 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:22:11.732 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:22:11.732 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:22:11.733 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:22:12.796 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:22:12.796 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:12.796 INFO:teuthology.orchestra.run.smithi135.stdout:Upgraded: 2024-03-29T11:22:12.796 INFO:teuthology.orchestra.run.smithi135.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:12.796 INFO:teuthology.orchestra.run.smithi135.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:12.797 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-29T11:22:12.797 INFO:teuthology.orchestra.run.smithi135.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:22:12.797 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:12.797 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:12.797 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:12.797 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:12.797 INFO:teuthology.orchestra.run.smithi135.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:22:12.797 INFO:teuthology.orchestra.run.smithi135.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:22:12.797 INFO:teuthology.orchestra.run.smithi135.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:22:12.797 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:12.797 INFO:teuthology.orchestra.run.smithi135.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:22:12.797 INFO:teuthology.orchestra.run.smithi135.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:22:12.797 INFO:teuthology.orchestra.run.smithi135.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:22:12.797 INFO:teuthology.orchestra.run.smithi135.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:12.797 INFO:teuthology.orchestra.run.smithi135.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:22:12.797 INFO:teuthology.orchestra.run.smithi135.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:12.798 INFO:teuthology.orchestra.run.smithi135.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:22:12.798 INFO:teuthology.orchestra.run.smithi135.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:22:12.798 INFO:teuthology.orchestra.run.smithi135.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:22:12.798 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:12.798 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:12.798 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:12.798 INFO:teuthology.orchestra.run.smithi135.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:22:12.798 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:12.798 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:12.798 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:12.798 INFO:teuthology.orchestra.run.smithi135.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:22:12.798 INFO:teuthology.orchestra.run.smithi135.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:22:12.798 INFO:teuthology.orchestra.run.smithi135.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:22:12.799 INFO:teuthology.orchestra.run.smithi135.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:22:12.799 INFO:teuthology.orchestra.run.smithi135.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:22:12.799 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:12.799 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-29T11:22:13.153 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-test 2024-03-29T11:22:13.709 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:00:34 ago on Fri 29 Mar 2024 11:21:39 AM UTC. 2024-03-29T11:22:13.876 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-29T11:22:13.876 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:22:13.877 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-03-29T11:22:13.877 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:22:13.877 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-29T11:22:13.877 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:22:13.877 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-03-29T11:22:13.877 INFO:teuthology.orchestra.run.smithi135.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:22:13.877 INFO:teuthology.orchestra.run.smithi135.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:22:13.877 INFO:teuthology.orchestra.run.smithi135.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:22:13.877 INFO:teuthology.orchestra.run.smithi135.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:22:13.877 INFO:teuthology.orchestra.run.smithi135.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:22:13.877 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:13.877 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-29T11:22:13.877 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:22:13.878 INFO:teuthology.orchestra.run.smithi135.stdout:Install 6 Packages 2024-03-29T11:22:13.878 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:13.878 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 48 M 2024-03-29T11:22:13.878 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 204 M 2024-03-29T11:22:13.878 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-29T11:22:14.303 INFO:teuthology.orchestra.run.smithi135.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-29T11:22:14.361 INFO:teuthology.orchestra.run.smithi135.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 300 kB 00:00 2024-03-29T11:22:14.386 INFO:teuthology.orchestra.run.smithi135.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-29T11:22:14.520 INFO:teuthology.orchestra.run.smithi135.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-29T11:22:14.645 INFO:teuthology.orchestra.run.smithi135.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 328 kB/s | 163 kB 00:00 2024-03-29T11:22:15.556 INFO:teuthology.orchestra.run.smithi194.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 3.4 MB/s | 47 MB 00:13 2024-03-29T11:22:15.559 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:15.559 INFO:teuthology.orchestra.run.smithi194.stdout:Total 3.4 MB/s | 48 MB 00:14 2024-03-29T11:22:15.573 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-29T11:22:15.594 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-29T11:22:15.594 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-29T11:22:15.976 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-29T11:22:15.977 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-29T11:22:16.554 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-29T11:22:16.623 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:22:16.662 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:22:16.702 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:22:16.748 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:22:16.775 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:22:18.195 INFO:teuthology.orchestra.run.smithi135.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 12 MB/s | 47 MB 00:04 2024-03-29T11:22:18.198 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:18.199 INFO:teuthology.orchestra.run.smithi135.stdout:Total 11 MB/s | 48 MB 00:04 2024-03-29T11:22:18.213 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-29T11:22:18.234 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-29T11:22:18.234 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-29T11:22:18.611 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-29T11:22:18.612 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-29T11:22:19.189 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-29T11:22:19.255 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:22:19.304 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:22:19.344 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:22:19.405 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:22:19.432 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:22:21.219 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:22:21.291 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:22:23.667 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:22:23.667 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:22:23.667 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:22:23.667 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:22:23.667 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:22:23.667 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:22:23.924 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:22:24.006 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:22:24.060 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:22:24.061 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:24.061 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-29T11:22:24.061 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:24.061 INFO:teuthology.orchestra.run.smithi194.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:22:24.061 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:24.061 INFO:teuthology.orchestra.run.smithi194.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:22:24.061 INFO:teuthology.orchestra.run.smithi194.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:22:24.061 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:22:24.061 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:24.061 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-29T11:22:24.349 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph 2024-03-29T11:22:24.915 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:51 ago on Fri 29 Mar 2024 11:21:33 AM UTC. 2024-03-29T11:22:25.071 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-29T11:22:25.074 INFO:teuthology.orchestra.run.smithi194.stdout:========================================================================================= 2024-03-29T11:22:25.074 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-29T11:22:25.074 INFO:teuthology.orchestra.run.smithi194.stdout:========================================================================================= 2024-03-29T11:22:25.074 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-29T11:22:25.074 INFO:teuthology.orchestra.run.smithi194.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:22:25.074 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-29T11:22:25.074 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:22:25.074 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:22:25.074 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:22:25.075 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:22:25.075 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:22:25.075 INFO:teuthology.orchestra.run.smithi194.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:22:25.075 INFO:teuthology.orchestra.run.smithi194.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:22:25.075 INFO:teuthology.orchestra.run.smithi194.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:22:25.075 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:22:25.075 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:22:25.075 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:22:25.075 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:22:25.075 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:22:25.075 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:22:25.075 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:22:25.075 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:22:25.075 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:22:25.076 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:22:25.076 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:22:25.076 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:22:25.076 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:22:25.076 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:22:25.076 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:22:25.076 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:22:25.076 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:22:25.076 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:22:25.076 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:22:25.076 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:22:25.077 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:22:25.077 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:22:25.077 INFO:teuthology.orchestra.run.smithi194.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:22:25.077 INFO:teuthology.orchestra.run.smithi194.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:22:25.077 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:22:25.077 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:22:25.077 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:22:25.077 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:25.077 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-29T11:22:25.077 INFO:teuthology.orchestra.run.smithi194.stdout:========================================================================================= 2024-03-29T11:22:25.077 INFO:teuthology.orchestra.run.smithi194.stdout:Install 36 Packages 2024-03-29T11:22:25.077 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:25.078 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 30 M 2024-03-29T11:22:25.078 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 107 M 2024-03-29T11:22:25.078 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-29T11:22:25.641 INFO:teuthology.orchestra.run.smithi194.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 29 kB/s | 6.6 kB 00:00 2024-03-29T11:22:25.792 INFO:teuthology.orchestra.run.smithi194.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 4.0 MB/s | 1.5 MB 00:00 2024-03-29T11:22:25.833 INFO:teuthology.orchestra.run.smithi194.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 5.1 MB/s | 2.1 MB 00:00 2024-03-29T11:22:25.900 INFO:teuthology.orchestra.run.smithi194.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 3.6 MB/s | 247 kB 00:00 2024-03-29T11:22:26.017 INFO:teuthology.orchestra.run.smithi194.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-29T11:22:26.059 INFO:teuthology.orchestra.run.smithi194.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T11:22:26.101 INFO:teuthology.orchestra.run.smithi194.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 756 kB/s | 32 kB 00:00 2024-03-29T11:22:26.135 INFO:teuthology.orchestra.run.smithi194.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:22:26.160 INFO:teuthology.orchestra.run.smithi194.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-29T11:22:26.202 INFO:teuthology.orchestra.run.smithi194.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-29T11:22:26.269 INFO:teuthology.orchestra.run.smithi194.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T11:22:26.319 INFO:teuthology.orchestra.run.smithi194.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-29T11:22:26.353 INFO:teuthology.orchestra.run.smithi194.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2024-03-29T11:22:26.386 INFO:teuthology.orchestra.run.smithi194.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-29T11:22:26.525 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:22:26.525 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:22:26.525 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:22:26.525 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:22:26.525 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:22:26.525 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:22:26.686 INFO:teuthology.orchestra.run.smithi194.stdout:(15/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 19 MB/s | 17 MB 00:00 2024-03-29T11:22:26.720 INFO:teuthology.orchestra.run.smithi194.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 374 kB/s | 125 kB 00:00 2024-03-29T11:22:26.804 INFO:teuthology.orchestra.run.smithi194.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 191 kB/s | 22 kB 00:00 2024-03-29T11:22:26.918 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:22:26.918 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:26.918 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-29T11:22:26.918 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:26.918 INFO:teuthology.orchestra.run.smithi135.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:22:26.918 INFO:teuthology.orchestra.run.smithi135.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:26.919 INFO:teuthology.orchestra.run.smithi135.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:22:26.919 INFO:teuthology.orchestra.run.smithi135.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:22:26.919 INFO:teuthology.orchestra.run.smithi135.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:22:26.919 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:26.919 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-29T11:22:27.054 INFO:teuthology.orchestra.run.smithi194.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 452 kB/s | 151 kB 00:00 2024-03-29T11:22:27.071 INFO:teuthology.orchestra.run.smithi194.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 163 kB/s | 43 kB 00:00 2024-03-29T11:22:27.163 INFO:teuthology.orchestra.run.smithi194.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-03-29T11:22:27.214 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph 2024-03-29T11:22:27.221 INFO:teuthology.orchestra.run.smithi194.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2024-03-29T11:22:27.289 INFO:teuthology.orchestra.run.smithi194.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.6 MB/s | 358 kB 00:00 2024-03-29T11:22:27.305 INFO:teuthology.orchestra.run.smithi194.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 211 kB/s | 18 kB 00:00 2024-03-29T11:22:27.389 INFO:teuthology.orchestra.run.smithi194.stdout:(24/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 2.7 MB/s | 4.7 MB 00:01 2024-03-29T11:22:27.406 INFO:teuthology.orchestra.run.smithi194.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 198 kB/s | 23 kB 00:00 2024-03-29T11:22:27.423 INFO:teuthology.orchestra.run.smithi194.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 166 kB/s | 19 kB 00:00 2024-03-29T11:22:27.481 INFO:teuthology.orchestra.run.smithi194.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 794 kB/s | 46 kB 00:00 2024-03-29T11:22:27.507 INFO:teuthology.orchestra.run.smithi194.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 943 kB/s | 79 kB 00:00 2024-03-29T11:22:27.574 INFO:teuthology.orchestra.run.smithi194.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-03-29T11:22:27.590 INFO:teuthology.orchestra.run.smithi194.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 199 kB/s | 16 kB 00:00 2024-03-29T11:22:27.607 INFO:teuthology.orchestra.run.smithi194.stdout:(31/36): python3-jaraco-text-3.2.0-6.el9.noarch 90 kB/s | 20 kB 00:00 2024-03-29T11:22:27.657 INFO:teuthology.orchestra.run.smithi194.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2024-03-29T11:22:27.683 INFO:teuthology.orchestra.run.smithi194.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 387 kB/s | 36 kB 00:00 2024-03-29T11:22:27.789 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:00:48 ago on Fri 29 Mar 2024 11:21:39 AM UTC. 2024-03-29T11:22:27.808 INFO:teuthology.orchestra.run.smithi194.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 160 kB/s | 20 kB 00:00 2024-03-29T11:22:27.833 INFO:teuthology.orchestra.run.smithi194.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.4 MB/s | 427 kB 00:00 2024-03-29T11:22:27.867 INFO:teuthology.orchestra.run.smithi194.stdout:(36/36): python3-webob-1.8.7-6.el9.noarch.rpm 886 kB/s | 230 kB 00:00 2024-03-29T11:22:27.871 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:27.872 INFO:teuthology.orchestra.run.smithi194.stdout:Total 11 MB/s | 30 MB 00:02 2024-03-29T11:22:27.896 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-29T11:22:27.938 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-29T11:22:27.938 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-29T11:22:27.943 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-29T11:22:27.945 INFO:teuthology.orchestra.run.smithi135.stdout:========================================================================================= 2024-03-29T11:22:27.945 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-03-29T11:22:27.945 INFO:teuthology.orchestra.run.smithi135.stdout:========================================================================================= 2024-03-29T11:22:27.946 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-29T11:22:27.946 INFO:teuthology.orchestra.run.smithi135.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:22:27.946 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-03-29T11:22:27.946 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:22:27.946 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:22:27.946 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:22:27.946 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:22:27.946 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:22:27.946 INFO:teuthology.orchestra.run.smithi135.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:22:27.946 INFO:teuthology.orchestra.run.smithi135.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:22:27.946 INFO:teuthology.orchestra.run.smithi135.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:22:27.947 INFO:teuthology.orchestra.run.smithi135.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:22:27.947 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:22:27.947 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:22:27.947 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:22:27.947 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:22:27.947 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:22:27.947 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:22:27.947 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:22:27.947 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:22:27.947 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:22:27.948 INFO:teuthology.orchestra.run.smithi135.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:22:27.948 INFO:teuthology.orchestra.run.smithi135.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:22:27.948 INFO:teuthology.orchestra.run.smithi135.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:22:27.948 INFO:teuthology.orchestra.run.smithi135.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:22:27.948 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:22:27.948 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:22:27.948 INFO:teuthology.orchestra.run.smithi135.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:22:27.948 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:22:27.948 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:22:27.948 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:22:27.948 INFO:teuthology.orchestra.run.smithi135.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:22:27.948 INFO:teuthology.orchestra.run.smithi135.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:22:27.948 INFO:teuthology.orchestra.run.smithi135.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:22:27.948 INFO:teuthology.orchestra.run.smithi135.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:22:27.949 INFO:teuthology.orchestra.run.smithi135.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:22:27.949 INFO:teuthology.orchestra.run.smithi135.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:22:27.949 INFO:teuthology.orchestra.run.smithi135.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:22:27.949 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:27.949 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-29T11:22:27.949 INFO:teuthology.orchestra.run.smithi135.stdout:========================================================================================= 2024-03-29T11:22:27.949 INFO:teuthology.orchestra.run.smithi135.stdout:Install 36 Packages 2024-03-29T11:22:27.949 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:27.950 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 30 M 2024-03-29T11:22:27.950 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 107 M 2024-03-29T11:22:27.950 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-29T11:22:28.340 INFO:teuthology.orchestra.run.smithi135.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 27 kB/s | 6.6 kB 00:00 2024-03-29T11:22:28.348 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-29T11:22:28.349 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-29T11:22:28.474 INFO:teuthology.orchestra.run.smithi135.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 4.0 MB/s | 1.5 MB 00:00 2024-03-29T11:22:28.516 INFO:teuthology.orchestra.run.smithi135.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 5.1 MB/s | 2.1 MB 00:00 2024-03-29T11:22:28.616 INFO:teuthology.orchestra.run.smithi135.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.4 MB/s | 247 kB 00:00 2024-03-29T11:22:28.758 INFO:teuthology.orchestra.run.smithi135.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-29T11:22:28.809 INFO:teuthology.orchestra.run.smithi135.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-29T11:22:28.861 INFO:teuthology.orchestra.run.smithi135.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 632 kB/s | 32 kB 00:00 2024-03-29T11:22:28.926 INFO:teuthology.orchestra.run.smithi135.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 711 kB/s | 47 kB 00:00 2024-03-29T11:22:29.001 INFO:teuthology.orchestra.run.smithi135.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 515 kB/s | 39 kB 00:00 2024-03-29T11:22:29.059 INFO:teuthology.orchestra.run.smithi135.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-29T11:22:29.152 INFO:teuthology.orchestra.run.smithi135.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 12 MB/s | 1.1 MB 00:00 2024-03-29T11:22:29.202 INFO:teuthology.orchestra.run.smithi135.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-29T11:22:29.235 INFO:teuthology.orchestra.run.smithi135.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T11:22:29.277 INFO:teuthology.orchestra.run.smithi135.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-29T11:22:29.528 INFO:teuthology.orchestra.run.smithi135.stdout:(15/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 16 MB/s | 17 MB 00:01 2024-03-29T11:22:29.553 INFO:teuthology.orchestra.run.smithi135.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 453 kB/s | 125 kB 00:00 2024-03-29T11:22:29.603 INFO:teuthology.orchestra.run.smithi135.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 297 kB/s | 22 kB 00:00 2024-03-29T11:22:29.704 INFO:teuthology.orchestra.run.smithi135.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-29T11:22:29.716 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-29T11:22:29.729 INFO:teuthology.orchestra.run.smithi135.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 347 kB/s | 43 kB 00:00 2024-03-29T11:22:29.769 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:22:29.771 INFO:teuthology.orchestra.run.smithi135.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-03-29T11:22:29.805 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:22:29.812 INFO:teuthology.orchestra.run.smithi135.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 256 kB/s | 11 kB 00:00 2024-03-29T11:22:29.838 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:22:29.846 INFO:teuthology.orchestra.run.smithi135.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.0 MB/s | 358 kB 00:00 2024-03-29T11:22:29.872 INFO:teuthology.orchestra.run.smithi135.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 301 kB/s | 18 kB 00:00 2024-03-29T11:22:29.897 INFO:teuthology.orchestra.run.smithi135.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-03-29T11:22:29.910 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:22:29.922 INFO:teuthology.orchestra.run.smithi135.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 384 kB/s | 19 kB 00:00 2024-03-29T11:22:29.945 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:22:29.947 INFO:teuthology.orchestra.run.smithi135.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 389 kB/s | 20 kB 00:00 2024-03-29T11:22:29.973 INFO:teuthology.orchestra.run.smithi135.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 924 kB/s | 46 kB 00:00 2024-03-29T11:22:29.978 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:22:29.998 INFO:teuthology.orchestra.run.smithi135.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-03-29T11:22:30.032 INFO:teuthology.orchestra.run.smithi135.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.6 MB/s | 272 kB 00:00 2024-03-29T11:22:30.057 INFO:teuthology.orchestra.run.smithi135.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 279 kB/s | 16 kB 00:00 2024-03-29T11:22:30.070 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:22:30.082 INFO:teuthology.orchestra.run.smithi135.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-03-29T11:22:30.108 INFO:teuthology.orchestra.run.smithi135.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-03-29T11:22:30.133 INFO:teuthology.orchestra.run.smithi135.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-29T11:22:30.140 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:22:30.166 INFO:teuthology.orchestra.run.smithi135.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 7.1 MB/s | 427 kB 00:00 2024-03-29T11:22:30.189 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:22:30.192 INFO:teuthology.orchestra.run.smithi135.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2024-03-29T11:22:30.261 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:22:30.315 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:22:30.353 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:22:30.402 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:22:30.456 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:22:30.641 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:22:30.690 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:22:30.720 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:22:30.755 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:22:30.785 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:22:30.861 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:22:30.895 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:22:30.929 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:22:30.969 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:22:31.004 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:22:31.038 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:22:31.120 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:22:31.161 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:22:31.233 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:22:31.369 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:22:31.486 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:22:31.643 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:22:31.666 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:22:31.691 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:22:31.691 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:22:31.691 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:22:31.691 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:22:31.691 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:22:31.692 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:32.117 INFO:teuthology.orchestra.run.smithi135.stdout:(36/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 1.2 MB/s | 4.7 MB 00:03 2024-03-29T11:22:32.121 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:32.121 INFO:teuthology.orchestra.run.smithi135.stdout:Total 7.1 MB/s | 30 MB 00:04 2024-03-29T11:22:32.147 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-29T11:22:32.189 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-29T11:22:32.189 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-29T11:22:32.596 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-29T11:22:32.596 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-29T11:22:33.125 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:22:33.151 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:22:33.151 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:22:33.152 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:22:33.152 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:22:33.152 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:22:33.152 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:33.592 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:22:33.615 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:22:33.616 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:22:33.616 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:22:33.616 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:22:33.616 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:22:33.616 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:33.826 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:22:33.849 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:22:33.849 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:22:33.849 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:22:33.849 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:22:33.849 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:22:33.849 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:33.917 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:22:34.038 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-29T11:22:34.111 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:22:34.148 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:22:34.180 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:22:34.251 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:22:34.295 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:22:34.337 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:22:34.429 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:22:34.499 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:22:34.548 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:22:34.620 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:22:34.674 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:22:34.712 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:22:34.769 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:22:34.825 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:22:35.011 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:22:35.058 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:22:35.096 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:22:35.131 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:22:35.162 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:22:35.238 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:22:35.271 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:22:35.305 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:22:35.354 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:22:35.397 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:22:35.432 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:22:35.522 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:22:35.562 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:22:35.634 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:22:35.685 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:22:35.685 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:22:35.685 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:22:35.685 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:22:35.685 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:22:35.685 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:22:35.685 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:22:35.685 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:22:35.685 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:22:35.685 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:22:35.685 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:22:35.686 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:22:35.686 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:22:35.686 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:22:35.686 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:22:35.686 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:22:35.686 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:22:35.686 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:22:35.686 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:22:35.686 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:22:35.686 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:22:35.686 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:22:35.686 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:22:35.686 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:22:35.686 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:22:35.687 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:22:35.687 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:22:35.687 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:22:35.687 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:22:35.687 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:22:35.687 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:22:35.688 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:22:35.688 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:22:35.688 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:22:35.688 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:22:35.688 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:22:35.773 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:22:35.907 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:22:36.063 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:22:36.101 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:22:36.126 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:22:36.127 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:22:36.127 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:22:36.127 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:22:36.127 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:22:36.127 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:36.555 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:22:36.556 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:36.556 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-29T11:22:36.556 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:36.556 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:36.556 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:36.556 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:36.556 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:36.556 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:36.556 INFO:teuthology.orchestra.run.smithi194.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:22:36.556 INFO:teuthology.orchestra.run.smithi194.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:22:36.556 INFO:teuthology.orchestra.run.smithi194.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:22:36.556 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:22:36.556 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:22:36.557 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:22:36.557 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:22:36.557 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:22:36.557 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:22:36.557 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:22:36.557 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:22:36.557 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:22:36.557 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:22:36.557 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:22:36.557 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:22:36.557 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:22:36.557 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:22:36.557 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:22:36.557 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:22:36.558 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:22:36.558 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:22:36.558 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:22:36.558 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:22:36.558 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:22:36.558 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:22:36.558 INFO:teuthology.orchestra.run.smithi194.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:22:36.558 INFO:teuthology.orchestra.run.smithi194.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:22:36.558 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:22:36.558 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:22:36.558 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:22:36.558 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:36.558 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-29T11:22:36.892 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-base 2024-03-29T11:22:37.464 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:04 ago on Fri 29 Mar 2024 11:21:33 AM UTC. 2024-03-29T11:22:37.562 INFO:teuthology.orchestra.run.smithi194.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:22:37.577 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:22:37.604 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:22:37.604 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:22:37.604 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:22:37.604 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:22:37.604 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:22:37.604 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:37.614 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-29T11:22:37.615 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-29T11:22:37.615 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-29T11:22:37.675 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install cephadm 2024-03-29T11:22:38.044 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:22:38.068 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:22:38.068 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:22:38.068 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:22:38.068 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:22:38.068 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:22:38.068 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:38.238 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:05 ago on Fri 29 Mar 2024 11:21:33 AM UTC. 2024-03-29T11:22:38.278 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:22:38.302 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:22:38.302 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:22:38.302 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:22:38.302 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:22:38.302 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:22:38.302 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:38.368 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:22:38.387 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-29T11:22:38.387 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:22:38.387 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-29T11:22:38.387 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:22:38.387 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-29T11:22:38.387 INFO:teuthology.orchestra.run.smithi194.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:22:38.387 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:38.387 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-29T11:22:38.388 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:22:38.388 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-29T11:22:38.388 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:38.388 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 747 k 2024-03-29T11:22:38.388 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 754 k 2024-03-29T11:22:38.388 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-29T11:22:39.089 INFO:teuthology.orchestra.run.smithi194.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.0 MB/s | 747 kB 00:00 2024-03-29T11:22:39.090 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:39.090 INFO:teuthology.orchestra.run.smithi194.stdout:Total 1.0 MB/s | 747 kB 00:00 2024-03-29T11:22:39.090 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-29T11:22:39.095 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-29T11:22:39.095 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-29T11:22:39.101 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-29T11:22:39.101 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-29T11:22:39.177 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-29T11:22:39.655 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:22:39.743 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:22:40.099 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:22:40.241 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:22:40.241 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:22:40.242 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:22:40.242 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:22:40.242 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:22:40.242 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:22:40.242 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:22:40.242 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:22:40.242 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:22:40.242 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:22:40.242 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:22:40.242 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:22:40.242 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:22:40.242 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:22:40.242 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:22:40.243 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:22:40.243 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:22:40.243 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:22:40.243 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:22:40.243 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:22:40.243 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:22:40.243 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:22:40.243 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:22:40.243 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:22:40.243 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:22:40.243 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:22:40.244 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:22:40.244 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:22:40.244 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:22:40.244 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:22:40.245 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:22:40.245 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:22:40.245 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:22:40.245 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:22:40.245 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:22:40.245 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:22:40.394 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:22:40.395 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:40.395 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-29T11:22:40.395 INFO:teuthology.orchestra.run.smithi194.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:40.395 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:40.395 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-29T11:22:40.611 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:22:41.181 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:08 ago on Fri 29 Mar 2024 11:21:33 AM UTC. 2024-03-29T11:22:41.217 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:22:41.217 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:41.217 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-29T11:22:41.217 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:41.217 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:41.217 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:41.217 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:41.217 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:41.217 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:41.217 INFO:teuthology.orchestra.run.smithi135.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:22:41.217 INFO:teuthology.orchestra.run.smithi135.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:22:41.218 INFO:teuthology.orchestra.run.smithi135.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:22:41.218 INFO:teuthology.orchestra.run.smithi135.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:22:41.218 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:22:41.218 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:22:41.218 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:22:41.218 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:22:41.218 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:22:41.218 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:22:41.218 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:22:41.218 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:22:41.218 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:22:41.218 INFO:teuthology.orchestra.run.smithi135.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:22:41.218 INFO:teuthology.orchestra.run.smithi135.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:22:41.219 INFO:teuthology.orchestra.run.smithi135.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:22:41.219 INFO:teuthology.orchestra.run.smithi135.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:22:41.219 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:22:41.219 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:22:41.219 INFO:teuthology.orchestra.run.smithi135.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:22:41.219 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:22:41.219 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:22:41.219 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:22:41.219 INFO:teuthology.orchestra.run.smithi135.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:22:41.219 INFO:teuthology.orchestra.run.smithi135.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:22:41.219 INFO:teuthology.orchestra.run.smithi135.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:22:41.219 INFO:teuthology.orchestra.run.smithi135.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:22:41.219 INFO:teuthology.orchestra.run.smithi135.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:22:41.219 INFO:teuthology.orchestra.run.smithi135.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:22:41.219 INFO:teuthology.orchestra.run.smithi135.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:22:41.219 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:41.220 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-29T11:22:41.335 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-29T11:22:41.336 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:22:41.336 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-29T11:22:41.336 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:22:41.336 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-29T11:22:41.337 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:22:41.337 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:41.337 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-29T11:22:41.337 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:22:41.337 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-29T11:22:41.337 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:41.337 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 145 k 2024-03-29T11:22:41.337 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 436 k 2024-03-29T11:22:41.337 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-29T11:22:41.562 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-base 2024-03-29T11:22:41.569 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 627 kB/s | 145 kB 00:00 2024-03-29T11:22:41.569 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:41.570 INFO:teuthology.orchestra.run.smithi194.stdout:Total 624 kB/s | 145 kB 00:00 2024-03-29T11:22:41.570 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-29T11:22:41.577 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-29T11:22:41.577 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-29T11:22:41.631 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-29T11:22:41.631 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-29T11:22:41.766 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-29T11:22:41.828 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:22:41.853 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:22:41.853 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:22:41.854 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:22:41.854 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:42.132 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:03 ago on Fri 29 Mar 2024 11:21:39 AM UTC. 2024-03-29T11:22:42.230 INFO:teuthology.orchestra.run.smithi135.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:22:42.282 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-29T11:22:42.283 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-03-29T11:22:42.283 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-29T11:22:42.357 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install cephadm 2024-03-29T11:22:42.833 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:22:42.834 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:42.834 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-29T11:22:42.834 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:42.834 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:42.834 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-29T11:22:42.924 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:03 ago on Fri 29 Mar 2024 11:21:39 AM UTC. 2024-03-29T11:22:43.064 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr 2024-03-29T11:22:43.075 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-29T11:22:43.075 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:22:43.075 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-03-29T11:22:43.076 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:22:43.076 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-29T11:22:43.076 INFO:teuthology.orchestra.run.smithi135.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:22:43.076 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:43.076 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-29T11:22:43.076 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:22:43.076 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-03-29T11:22:43.076 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:43.076 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 747 k 2024-03-29T11:22:43.076 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 754 k 2024-03-29T11:22:43.076 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-29T11:22:43.643 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:10 ago on Fri 29 Mar 2024 11:21:33 AM UTC. 2024-03-29T11:22:43.686 INFO:teuthology.orchestra.run.smithi135.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.2 MB/s | 747 kB 00:00 2024-03-29T11:22:43.687 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:43.687 INFO:teuthology.orchestra.run.smithi135.stdout:Total 1.2 MB/s | 747 kB 00:00 2024-03-29T11:22:43.687 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-29T11:22:43.693 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-29T11:22:43.693 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-29T11:22:43.698 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-29T11:22:43.699 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-29T11:22:43.741 INFO:teuthology.orchestra.run.smithi194.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:22:43.775 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-29T11:22:43.793 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-29T11:22:43.794 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-29T11:22:43.794 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-29T11:22:43.858 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:22:44.219 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:22:44.307 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:22:44.420 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:11 ago on Fri 29 Mar 2024 11:21:33 AM UTC. 2024-03-29T11:22:44.572 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-29T11:22:44.573 INFO:teuthology.orchestra.run.smithi194.stdout:====================================================================================== 2024-03-29T11:22:44.573 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-29T11:22:44.573 INFO:teuthology.orchestra.run.smithi194.stdout:====================================================================================== 2024-03-29T11:22:44.573 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-29T11:22:44.573 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:22:44.573 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-29T11:22:44.573 INFO:teuthology.orchestra.run.smithi194.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:22:44.574 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:22:44.574 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:22:44.574 INFO:teuthology.orchestra.run.smithi194.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:22:44.574 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:22:44.574 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:22:44.574 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:22:44.574 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:22:44.574 INFO:teuthology.orchestra.run.smithi194.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:22:44.574 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:22:44.574 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:22:44.574 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:44.574 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-29T11:22:44.574 INFO:teuthology.orchestra.run.smithi194.stdout:====================================================================================== 2024-03-29T11:22:44.575 INFO:teuthology.orchestra.run.smithi194.stdout:Install 12 Packages 2024-03-29T11:22:44.575 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:44.575 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 6.9 M 2024-03-29T11:22:44.575 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 26 M 2024-03-29T11:22:44.575 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-29T11:22:44.689 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:22:44.978 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:22:44.978 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:44.978 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-29T11:22:44.978 INFO:teuthology.orchestra.run.smithi135.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:44.978 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:44.978 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-29T11:22:45.081 INFO:teuthology.orchestra.run.smithi194.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 81 kB/s | 15 kB 00:00 2024-03-29T11:22:45.140 INFO:teuthology.orchestra.run.smithi194.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 126 kB/s | 31 kB 00:00 2024-03-29T11:22:45.199 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:22:45.240 INFO:teuthology.orchestra.run.smithi194.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-29T11:22:45.257 INFO:teuthology.orchestra.run.smithi194.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 238 kB 00:00 2024-03-29T11:22:45.449 INFO:teuthology.orchestra.run.smithi194.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.1 MB/s | 863 kB 00:00 2024-03-29T11:22:45.699 INFO:teuthology.orchestra.run.smithi194.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 78 kB/s | 19 kB 00:00 2024-03-29T11:22:45.733 INFO:teuthology.orchestra.run.smithi194.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-03-29T11:22:45.774 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:06 ago on Fri 29 Mar 2024 11:21:39 AM UTC. 2024-03-29T11:22:45.816 INFO:teuthology.orchestra.run.smithi194.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.7 MB/s | 144 kB 00:00 2024-03-29T11:22:45.908 INFO:teuthology.orchestra.run.smithi194.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 336 kB/s | 31 kB 00:00 2024-03-29T11:22:45.923 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-29T11:22:45.923 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:22:45.923 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-03-29T11:22:45.923 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:22:45.924 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-29T11:22:45.924 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:22:45.924 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:45.924 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-29T11:22:45.924 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:22:45.924 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-03-29T11:22:45.924 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:45.925 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 145 k 2024-03-29T11:22:45.925 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 436 k 2024-03-29T11:22:45.925 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-29T11:22:45.992 INFO:teuthology.orchestra.run.smithi194.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-29T11:22:46.143 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 667 kB/s | 145 kB 00:00 2024-03-29T11:22:46.144 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:46.144 INFO:teuthology.orchestra.run.smithi135.stdout:Total 663 kB/s | 145 kB 00:00 2024-03-29T11:22:46.144 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-29T11:22:46.148 INFO:teuthology.orchestra.run.smithi194.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.5 MB/s | 2.0 MB 00:00 2024-03-29T11:22:46.151 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-29T11:22:46.151 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-29T11:22:46.204 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-29T11:22:46.204 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-29T11:22:46.347 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-29T11:22:46.419 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:22:46.443 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:22:46.443 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:22:46.444 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:22:46.444 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:46.509 INFO:teuthology.orchestra.run.smithi194.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 1.1 MB/s | 1.8 MB 00:01 2024-03-29T11:22:46.513 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:46.513 INFO:teuthology.orchestra.run.smithi194.stdout:Total 3.5 MB/s | 6.9 MB 00:01 2024-03-29T11:22:46.532 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-29T11:22:46.550 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-29T11:22:46.550 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-29T11:22:46.718 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-29T11:22:46.718 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-29T11:22:47.241 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-29T11:22:47.309 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:22:47.344 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:22:47.358 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:22:47.358 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:47.358 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-29T11:22:47.358 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:22:47.358 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:47.358 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-29T11:22:47.390 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:22:47.429 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:22:47.531 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:22:47.629 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr 2024-03-29T11:22:47.653 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:22:47.747 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:22:47.792 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:22:47.820 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:22:47.852 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:22:48.090 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:22:48.127 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:22:48.208 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:09 ago on Fri 29 Mar 2024 11:21:39 AM UTC. 2024-03-29T11:22:48.307 INFO:teuthology.orchestra.run.smithi135.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:22:48.359 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-29T11:22:48.360 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-03-29T11:22:48.360 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-29T11:22:48.441 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:22:48.915 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:22:48.915 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:22:48.915 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:22:48.916 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:22:48.916 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:22:48.916 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:22:48.916 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:22:48.916 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:22:48.916 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:22:48.916 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:22:48.916 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:22:48.916 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:22:49.003 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:10 ago on Fri 29 Mar 2024 11:21:39 AM UTC. 2024-03-29T11:22:49.155 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-29T11:22:49.156 INFO:teuthology.orchestra.run.smithi135.stdout:====================================================================================== 2024-03-29T11:22:49.156 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-03-29T11:22:49.156 INFO:teuthology.orchestra.run.smithi135.stdout:====================================================================================== 2024-03-29T11:22:49.156 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-29T11:22:49.156 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:22:49.156 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-03-29T11:22:49.156 INFO:teuthology.orchestra.run.smithi135.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:22:49.156 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:22:49.156 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:22:49.157 INFO:teuthology.orchestra.run.smithi135.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:22:49.157 INFO:teuthology.orchestra.run.smithi135.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:22:49.157 INFO:teuthology.orchestra.run.smithi135.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:22:49.157 INFO:teuthology.orchestra.run.smithi135.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:22:49.157 INFO:teuthology.orchestra.run.smithi135.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:22:49.157 INFO:teuthology.orchestra.run.smithi135.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:22:49.157 INFO:teuthology.orchestra.run.smithi135.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:22:49.157 INFO:teuthology.orchestra.run.smithi135.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:22:49.157 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:49.157 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-29T11:22:49.157 INFO:teuthology.orchestra.run.smithi135.stdout:====================================================================================== 2024-03-29T11:22:49.157 INFO:teuthology.orchestra.run.smithi135.stdout:Install 12 Packages 2024-03-29T11:22:49.157 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:49.158 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 6.9 M 2024-03-29T11:22:49.158 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 26 M 2024-03-29T11:22:49.158 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-29T11:22:49.268 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:22:49.268 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:49.269 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-29T11:22:49.269 INFO:teuthology.orchestra.run.smithi194.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:22:49.269 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:49.269 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:49.269 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:49.269 INFO:teuthology.orchestra.run.smithi194.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:22:49.269 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:22:49.269 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:22:49.269 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:22:49.269 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:22:49.269 INFO:teuthology.orchestra.run.smithi194.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:22:49.269 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:22:49.269 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:22:49.269 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:49.270 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-29T11:22:49.485 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:22:49.780 INFO:teuthology.orchestra.run.smithi135.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 76 kB/s | 15 kB 00:00 2024-03-29T11:22:49.805 INFO:teuthology.orchestra.run.smithi135.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 138 kB/s | 31 kB 00:00 2024-03-29T11:22:50.046 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:17 ago on Fri 29 Mar 2024 11:21:33 AM UTC. 2024-03-29T11:22:50.072 INFO:teuthology.orchestra.run.smithi135.stdout:(3/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.6 MB/s | 1.8 MB 00:00 2024-03-29T11:22:50.200 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-29T11:22:50.201 INFO:teuthology.orchestra.run.smithi194.stdout:============================================================================================ 2024-03-29T11:22:50.201 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-29T11:22:50.201 INFO:teuthology.orchestra.run.smithi194.stdout:============================================================================================ 2024-03-29T11:22:50.201 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-29T11:22:50.202 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:22:50.202 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-29T11:22:50.202 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:22:50.202 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:22:50.202 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:22:50.202 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:22:50.202 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:22:50.202 INFO:teuthology.orchestra.run.smithi194.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:22:50.202 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:22:50.202 INFO:teuthology.orchestra.run.smithi194.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:22:50.202 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:22:50.202 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:22:50.202 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:22:50.202 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:50.203 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-29T11:22:50.203 INFO:teuthology.orchestra.run.smithi194.stdout:============================================================================================ 2024-03-29T11:22:50.203 INFO:teuthology.orchestra.run.smithi194.stdout:Install 12 Packages 2024-03-29T11:22:50.203 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:22:50.203 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 37 M 2024-03-29T11:22:50.203 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 209 M 2024-03-29T11:22:50.203 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-29T11:22:50.256 INFO:teuthology.orchestra.run.smithi135.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 528 kB/s | 238 kB 00:00 2024-03-29T11:22:50.565 INFO:teuthology.orchestra.run.smithi135.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.3 MB/s | 1.0 MB 00:00 2024-03-29T11:22:50.706 INFO:teuthology.orchestra.run.smithi194.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 148 kB/s | 32 kB 00:00 2024-03-29T11:22:50.758 INFO:teuthology.orchestra.run.smithi135.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-03-29T11:22:50.772 INFO:teuthology.orchestra.run.smithi194.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 257 kB/s | 17 kB 00:00 2024-03-29T11:22:50.782 INFO:teuthology.orchestra.run.smithi135.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 90 kB/s | 19 kB 00:00 2024-03-29T11:22:50.807 INFO:teuthology.orchestra.run.smithi135.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.1 MB/s | 863 kB 00:00 2024-03-29T11:22:50.856 INFO:teuthology.orchestra.run.smithi194.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 409 kB/s | 34 kB 00:00 2024-03-29T11:22:51.082 INFO:teuthology.orchestra.run.smithi135.stdout:(9/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 6.3 MB/s | 2.0 MB 00:00 2024-03-29T11:22:51.090 INFO:teuthology.orchestra.run.smithi194.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.0 MB/s | 3.0 MB 00:00 2024-03-29T11:22:51.116 INFO:teuthology.orchestra.run.smithi135.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 432 kB/s | 144 kB 00:00 2024-03-29T11:22:51.141 INFO:teuthology.orchestra.run.smithi135.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 92 kB/s | 31 kB 00:00 2024-03-29T11:22:51.166 INFO:teuthology.orchestra.run.smithi135.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-29T11:22:51.171 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:51.171 INFO:teuthology.orchestra.run.smithi135.stdout:Total 3.4 MB/s | 6.9 MB 00:02 2024-03-29T11:22:51.191 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-29T11:22:51.208 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-29T11:22:51.208 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-29T11:22:51.391 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-29T11:22:51.392 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-29T11:22:51.440 INFO:teuthology.orchestra.run.smithi194.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 8.1 MB/s | 4.7 MB 00:00 2024-03-29T11:22:51.507 INFO:teuthology.orchestra.run.smithi194.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 4.8 MB 00:00 2024-03-29T11:22:51.615 INFO:teuthology.orchestra.run.smithi194.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.2 MB/s | 217 kB 00:00 2024-03-29T11:22:51.913 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-29T11:22:52.000 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:22:52.024 INFO:teuthology.orchestra.run.smithi194.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-03-29T11:22:52.044 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:22:52.099 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:22:52.130 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:22:52.232 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:22:52.361 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:22:52.464 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:22:52.509 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:22:52.537 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:22:52.569 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:22:52.811 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:22:52.844 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:22:53.441 INFO:teuthology.orchestra.run.smithi194.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 140 kB/s | 195 kB 00:01 2024-03-29T11:22:53.686 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:22:53.687 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:22:53.687 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:22:53.687 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:22:53.687 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:22:53.687 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:22:53.687 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:22:53.687 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:22:53.687 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:22:53.687 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:22:53.687 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:22:53.687 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:22:53.783 INFO:teuthology.orchestra.run.smithi194.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 719 kB/s | 245 kB 00:00 2024-03-29T11:22:54.008 INFO:teuthology.orchestra.run.smithi194.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 336 kB/s | 803 kB 00:02 2024-03-29T11:22:54.034 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:22:54.034 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:54.034 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-29T11:22:54.034 INFO:teuthology.orchestra.run.smithi135.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:22:54.034 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:54.034 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:54.034 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:22:54.035 INFO:teuthology.orchestra.run.smithi135.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:22:54.035 INFO:teuthology.orchestra.run.smithi135.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:22:54.035 INFO:teuthology.orchestra.run.smithi135.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:22:54.035 INFO:teuthology.orchestra.run.smithi135.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:22:54.035 INFO:teuthology.orchestra.run.smithi135.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:22:54.035 INFO:teuthology.orchestra.run.smithi135.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:22:54.035 INFO:teuthology.orchestra.run.smithi135.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:22:54.035 INFO:teuthology.orchestra.run.smithi135.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:22:54.035 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:54.035 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-29T11:22:54.298 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:22:54.878 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:15 ago on Fri 29 Mar 2024 11:21:39 AM UTC. 2024-03-29T11:22:55.041 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-29T11:22:55.042 INFO:teuthology.orchestra.run.smithi135.stdout:============================================================================================ 2024-03-29T11:22:55.042 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-03-29T11:22:55.042 INFO:teuthology.orchestra.run.smithi135.stdout:============================================================================================ 2024-03-29T11:22:55.042 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-29T11:22:55.042 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:22:55.042 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-03-29T11:22:55.042 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:22:55.042 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:22:55.042 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:22:55.043 INFO:teuthology.orchestra.run.smithi135.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:22:55.043 INFO:teuthology.orchestra.run.smithi135.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:22:55.043 INFO:teuthology.orchestra.run.smithi135.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:22:55.043 INFO:teuthology.orchestra.run.smithi135.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:22:55.043 INFO:teuthology.orchestra.run.smithi135.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:22:55.043 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:22:55.043 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:22:55.043 INFO:teuthology.orchestra.run.smithi135.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:22:55.043 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:55.043 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-29T11:22:55.043 INFO:teuthology.orchestra.run.smithi135.stdout:============================================================================================ 2024-03-29T11:22:55.043 INFO:teuthology.orchestra.run.smithi135.stdout:Install 12 Packages 2024-03-29T11:22:55.044 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:22:55.044 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 37 M 2024-03-29T11:22:55.044 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 209 M 2024-03-29T11:22:55.044 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-29T11:22:55.424 INFO:teuthology.orchestra.run.smithi135.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 225 kB/s | 32 kB 00:00 2024-03-29T11:22:55.466 INFO:teuthology.orchestra.run.smithi135.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 426 kB/s | 17 kB 00:00 2024-03-29T11:22:55.549 INFO:teuthology.orchestra.run.smithi135.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-29T11:22:55.574 INFO:teuthology.orchestra.run.smithi135.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 320 kB/s | 34 kB 00:00 2024-03-29T11:22:55.792 INFO:teuthology.orchestra.run.smithi135.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-29T11:22:55.859 INFO:teuthology.orchestra.run.smithi135.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-03-29T11:22:55.884 INFO:teuthology.orchestra.run.smithi135.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T11:22:56.034 INFO:teuthology.orchestra.run.smithi135.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-14 9.8 MB/s | 7.4 MB 00:00 2024-03-29T11:22:56.151 INFO:teuthology.orchestra.run.smithi135.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 195 kB 00:00 2024-03-29T11:22:56.235 INFO:teuthology.orchestra.run.smithi135.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.2 MB/s | 803 kB 00:00 2024-03-29T11:22:56.310 INFO:teuthology.orchestra.run.smithi135.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-03-29T11:22:56.469 INFO:teuthology.orchestra.run.smithi135.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-03-29T11:22:56.471 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:56.471 INFO:teuthology.orchestra.run.smithi135.stdout:Total 26 MB/s | 37 MB 00:01 2024-03-29T11:22:56.511 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-29T11:22:56.535 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-29T11:22:56.535 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-29T11:22:56.894 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-29T11:22:56.895 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-29T11:22:57.598 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-29T11:22:57.658 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:22:57.698 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:22:57.790 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:22:57.834 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:22:58.012 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:22:58.042 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:22:58.130 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:22:58.803 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:22:58.921 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:22:59.274 INFO:teuthology.orchestra.run.smithi194.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 862 kB/s | 7.4 MB 00:08 2024-03-29T11:22:59.278 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:59.278 INFO:teuthology.orchestra.run.smithi194.stdout:Total 4.1 MB/s | 37 MB 00:09 2024-03-29T11:22:59.321 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-29T11:22:59.348 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-29T11:22:59.348 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-29T11:22:59.709 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-29T11:22:59.710 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-29T11:23:00.164 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:23:00.424 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-29T11:23:00.489 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:23:00.520 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:23:00.613 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:23:00.648 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:23:00.816 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:23:00.839 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:23:00.915 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:23:01.191 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:23:01.216 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:23:01.567 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:23:01.668 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:23:02.903 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:23:03.936 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:23:04.167 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:23:04.168 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:23:04.168 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:23:04.168 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:23:04.168 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:23:04.168 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:23:04.168 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:23:04.168 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:23:04.168 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:23:04.168 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:23:04.168 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:23:04.168 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:23:04.179 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:23:04.569 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:23:04.569 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:04.569 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-29T11:23:04.569 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:23:04.569 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:23:04.569 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:23:04.570 INFO:teuthology.orchestra.run.smithi135.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:23:04.570 INFO:teuthology.orchestra.run.smithi135.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:23:04.570 INFO:teuthology.orchestra.run.smithi135.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:23:04.570 INFO:teuthology.orchestra.run.smithi135.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:23:04.570 INFO:teuthology.orchestra.run.smithi135.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:23:04.570 INFO:teuthology.orchestra.run.smithi135.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:23:04.570 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:23:04.570 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:23:04.570 INFO:teuthology.orchestra.run.smithi135.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:23:04.570 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:04.570 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-29T11:23:05.001 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:23:05.581 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:26 ago on Fri 29 Mar 2024 11:21:39 AM UTC. 2024-03-29T11:23:05.734 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-29T11:23:05.735 INFO:teuthology.orchestra.run.smithi135.stdout:======================================================================================== 2024-03-29T11:23:05.735 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-03-29T11:23:05.735 INFO:teuthology.orchestra.run.smithi135.stdout:======================================================================================== 2024-03-29T11:23:05.735 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-29T11:23:05.735 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:23:05.735 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-03-29T11:23:05.735 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:23:05.735 INFO:teuthology.orchestra.run.smithi135.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:23:05.736 INFO:teuthology.orchestra.run.smithi135.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:23:05.736 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:23:05.736 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:23:05.736 INFO:teuthology.orchestra.run.smithi135.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:23:05.736 INFO:teuthology.orchestra.run.smithi135.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:23:05.736 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:23:05.736 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:23:05.736 INFO:teuthology.orchestra.run.smithi135.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:23:05.736 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:23:05.736 INFO:teuthology.orchestra.run.smithi135.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:23:05.736 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:05.736 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-29T11:23:05.736 INFO:teuthology.orchestra.run.smithi135.stdout:======================================================================================== 2024-03-29T11:23:05.737 INFO:teuthology.orchestra.run.smithi135.stdout:Install 13 Packages 2024-03-29T11:23:05.737 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:05.737 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 2.0 M 2024-03-29T11:23:05.737 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 26 M 2024-03-29T11:23:05.737 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-29T11:23:05.942 INFO:teuthology.orchestra.run.smithi135.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 190 kB/s | 24 kB 00:00 2024-03-29T11:23:05.967 INFO:teuthology.orchestra.run.smithi135.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 118 kB/s | 18 kB 00:00 2024-03-29T11:23:06.009 INFO:teuthology.orchestra.run.smithi135.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-29T11:23:06.034 INFO:teuthology.orchestra.run.smithi135.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-29T11:23:06.059 INFO:teuthology.orchestra.run.smithi135.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:23:06.085 INFO:teuthology.orchestra.run.smithi135.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-29T11:23:06.110 INFO:teuthology.orchestra.run.smithi135.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 176 kB/s | 52 kB 00:00 2024-03-29T11:23:06.335 INFO:teuthology.orchestra.run.smithi135.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 117 kB/s | 32 kB 00:00 2024-03-29T11:23:06.360 INFO:teuthology.orchestra.run.smithi135.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 54 kB/s | 15 kB 00:00 2024-03-29T11:23:06.374 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:23:06.374 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:23:06.374 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:23:06.374 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:23:06.374 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:23:06.374 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:23:06.374 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:23:06.374 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:23:06.375 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:23:06.375 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:23:06.375 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:23:06.375 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:23:06.519 INFO:teuthology.orchestra.run.smithi135.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 373 kB/s | 59 kB 00:00 2024-03-29T11:23:06.544 INFO:teuthology.orchestra.run.smithi135.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 467 kB/s | 203 kB 00:00 2024-03-29T11:23:06.636 INFO:teuthology.orchestra.run.smithi135.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 768 kB/s | 90 kB 00:00 2024-03-29T11:23:06.678 INFO:teuthology.orchestra.run.smithi135.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.0 MB/s | 1.0 MB 00:00 2024-03-29T11:23:06.682 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:06.682 INFO:teuthology.orchestra.run.smithi135.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2024-03-29T11:23:06.683 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-29T11:23:06.695 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-29T11:23:06.695 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-29T11:23:06.788 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-29T11:23:06.789 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-29T11:23:06.861 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:23:06.861 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:06.862 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-29T11:23:06.862 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:23:06.862 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:23:06.862 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:23:06.862 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:23:06.862 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:23:06.862 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:23:06.862 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:23:06.862 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:23:06.862 INFO:teuthology.orchestra.run.smithi194.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:23:06.862 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:23:06.862 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:23:06.862 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:23:06.862 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:06.862 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-29T11:23:07.199 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:23:07.292 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-29T11:23:07.465 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:23:07.554 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:23:07.617 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:23:07.647 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:23:07.688 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:23:07.768 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Mar 2024 11:21:33 AM UTC. 2024-03-29T11:23:07.782 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:23:07.891 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:23:07.920 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-29T11:23:07.921 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================== 2024-03-29T11:23:07.921 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-29T11:23:07.921 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================== 2024-03-29T11:23:07.921 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-29T11:23:07.921 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:23:07.921 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-29T11:23:07.921 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:23:07.921 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:23:07.922 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:23:07.922 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:23:07.922 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:23:07.922 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:23:07.922 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:23:07.922 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:23:07.922 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:23:07.922 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:23:07.922 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:23:07.922 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:23:07.922 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:07.922 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-29T11:23:07.922 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================== 2024-03-29T11:23:07.922 INFO:teuthology.orchestra.run.smithi194.stdout:Install 13 Packages 2024-03-29T11:23:07.923 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:07.923 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 2.0 M 2024-03-29T11:23:07.923 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 26 M 2024-03-29T11:23:07.923 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-29T11:23:07.948 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:23:08.247 INFO:teuthology.orchestra.run.smithi194.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 136 kB/s | 18 kB 00:00 2024-03-29T11:23:08.272 INFO:teuthology.orchestra.run.smithi194.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 155 kB/s | 24 kB 00:00 2024-03-29T11:23:08.322 INFO:teuthology.orchestra.run.smithi194.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T11:23:08.348 INFO:teuthology.orchestra.run.smithi194.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T11:23:08.373 INFO:teuthology.orchestra.run.smithi194.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:23:08.398 INFO:teuthology.orchestra.run.smithi194.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2024-03-29T11:23:08.424 INFO:teuthology.orchestra.run.smithi194.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 169 kB/s | 52 kB 00:00 2024-03-29T11:23:08.669 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:23:08.682 INFO:teuthology.orchestra.run.smithi194.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 53 kB/s | 15 kB 00:00 2024-03-29T11:23:08.716 INFO:teuthology.orchestra.run.smithi194.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 94 kB/s | 32 kB 00:00 2024-03-29T11:23:08.720 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:23:08.785 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:23:08.836 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:23:08.841 INFO:teuthology.orchestra.run.smithi194.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 473 kB/s | 59 kB 00:00 2024-03-29T11:23:08.857 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:23:08.866 INFO:teuthology.orchestra.run.smithi194.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 458 kB/s | 203 kB 00:00 2024-03-29T11:23:08.966 INFO:teuthology.orchestra.run.smithi194.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.6 MB/s | 1.0 MB 00:00 2024-03-29T11:23:08.991 INFO:teuthology.orchestra.run.smithi194.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 598 kB/s | 90 kB 00:00 2024-03-29T11:23:08.995 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:08.996 INFO:teuthology.orchestra.run.smithi194.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-03-29T11:23:08.996 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-29T11:23:09.008 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-29T11:23:09.009 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-29T11:23:09.098 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-29T11:23:09.098 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-29T11:23:09.589 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:23:09.589 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:23:09.589 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:23:09.589 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:23:09.589 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:23:09.589 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:23:09.589 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:23:09.589 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:23:09.589 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:23:09.590 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:23:09.590 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:23:09.590 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:23:09.590 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:23:09.609 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-29T11:23:09.777 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:23:09.834 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:23:09.890 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:23:09.926 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:23:09.951 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:23:09.951 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:09.951 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-29T11:23:09.951 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:23:09.951 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:23:09.952 INFO:teuthology.orchestra.run.smithi135.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:23:09.952 INFO:teuthology.orchestra.run.smithi135.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:23:09.952 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:23:09.952 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:23:09.952 INFO:teuthology.orchestra.run.smithi135.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:23:09.952 INFO:teuthology.orchestra.run.smithi135.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:23:09.952 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:23:09.952 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:23:09.952 INFO:teuthology.orchestra.run.smithi135.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:23:09.952 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:23:09.952 INFO:teuthology.orchestra.run.smithi135.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:23:09.952 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:09.952 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-29T11:23:09.967 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:23:10.062 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:23:10.171 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:23:10.220 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:23:10.230 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:23:10.802 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:31 ago on Fri 29 Mar 2024 11:21:39 AM UTC. 2024-03-29T11:23:10.938 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:23:10.953 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-29T11:23:10.954 INFO:teuthology.orchestra.run.smithi135.stdout:======================================================================================== 2024-03-29T11:23:10.954 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-03-29T11:23:10.954 INFO:teuthology.orchestra.run.smithi135.stdout:======================================================================================== 2024-03-29T11:23:10.954 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-29T11:23:10.955 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:23:10.955 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-03-29T11:23:10.955 INFO:teuthology.orchestra.run.smithi135.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:23:10.955 INFO:teuthology.orchestra.run.smithi135.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:23:10.955 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:23:10.955 INFO:teuthology.orchestra.run.smithi135.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:23:10.955 INFO:teuthology.orchestra.run.smithi135.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:23:10.955 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:10.955 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-29T11:23:10.955 INFO:teuthology.orchestra.run.smithi135.stdout:======================================================================================== 2024-03-29T11:23:10.955 INFO:teuthology.orchestra.run.smithi135.stdout:Install 6 Packages 2024-03-29T11:23:10.955 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:10.956 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 6.8 M 2024-03-29T11:23:10.956 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 33 M 2024-03-29T11:23:10.956 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-29T11:23:10.983 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:23:11.016 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:23:11.065 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:23:11.086 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:23:11.272 INFO:teuthology.orchestra.run.smithi135.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-03-29T11:23:11.380 INFO:teuthology.orchestra.run.smithi135.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-29T11:23:11.514 INFO:teuthology.orchestra.run.smithi135.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 333 kB/s | 138 kB 00:00 2024-03-29T11:23:11.648 INFO:teuthology.orchestra.run.smithi135.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 237 kB/s | 58 kB 00:00 2024-03-29T11:23:11.706 INFO:teuthology.orchestra.run.smithi135.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.2 MB/s | 548 kB 00:00 2024-03-29T11:23:11.731 INFO:teuthology.orchestra.run.smithi135.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 236 kB/s | 51 kB 00:00 2024-03-29T11:23:11.734 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:11.734 INFO:teuthology.orchestra.run.smithi135.stdout:Total 8.7 MB/s | 6.8 MB 00:00 2024-03-29T11:23:11.746 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-29T11:23:11.756 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-29T11:23:11.756 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-29T11:23:11.844 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-29T11:23:11.845 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-29T11:23:11.865 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:23:11.865 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:23:11.865 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:23:11.865 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:23:11.865 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:23:11.865 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:23:11.865 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:23:11.866 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:23:11.866 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:23:11.866 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:23:11.866 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:23:11.866 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:23:11.866 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:23:12.148 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-29T11:23:12.224 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:23:12.224 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:12.224 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-29T11:23:12.224 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:23:12.224 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:23:12.224 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:23:12.224 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:23:12.224 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:23:12.224 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:23:12.224 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:23:12.224 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:23:12.225 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:23:12.225 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:23:12.225 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:23:12.225 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:23:12.225 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:23:12.225 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:12.225 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-29T11:23:12.311 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:23:12.357 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:23:12.449 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:23:12.871 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:23:12.942 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:23:13.013 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:23:13.018 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 11:21:33 AM UTC. 2024-03-29T11:23:13.045 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:23:13.170 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-29T11:23:13.171 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================== 2024-03-29T11:23:13.171 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-29T11:23:13.171 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================== 2024-03-29T11:23:13.171 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-29T11:23:13.171 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:23:13.171 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-29T11:23:13.171 INFO:teuthology.orchestra.run.smithi194.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:23:13.171 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:23:13.171 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:23:13.171 INFO:teuthology.orchestra.run.smithi194.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:23:13.171 INFO:teuthology.orchestra.run.smithi194.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:23:13.171 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:13.171 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-29T11:23:13.172 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================== 2024-03-29T11:23:13.172 INFO:teuthology.orchestra.run.smithi194.stdout:Install 6 Packages 2024-03-29T11:23:13.172 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:13.172 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 6.8 M 2024-03-29T11:23:13.172 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 33 M 2024-03-29T11:23:13.172 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-29T11:23:13.830 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:23:13.831 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:23:13.831 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:23:13.831 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:23:13.831 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:23:13.831 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:23:13.942 INFO:teuthology.orchestra.run.smithi194.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-03-29T11:23:14.139 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:23:14.139 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:14.139 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-29T11:23:14.139 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:23:14.139 INFO:teuthology.orchestra.run.smithi135.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:23:14.139 INFO:teuthology.orchestra.run.smithi135.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:23:14.139 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:23:14.139 INFO:teuthology.orchestra.run.smithi135.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:23:14.140 INFO:teuthology.orchestra.run.smithi135.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:23:14.140 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:14.140 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-29T11:23:14.150 INFO:teuthology.orchestra.run.smithi194.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-03-29T11:23:14.234 INFO:teuthology.orchestra.run.smithi194.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 278 kB/s | 138 kB 00:00 2024-03-29T11:23:14.268 INFO:teuthology.orchestra.run.smithi194.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2024-03-29T11:23:14.301 INFO:teuthology.orchestra.run.smithi194.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 384 kB/s | 58 kB 00:00 2024-03-29T11:23:14.384 INFO:teuthology.orchestra.run.smithi194.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 340 kB/s | 51 kB 00:00 2024-03-29T11:23:14.386 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:14.386 INFO:teuthology.orchestra.run.smithi194.stdout:Total 5.6 MB/s | 6.8 MB 00:01 2024-03-29T11:23:14.395 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-fuse 2024-03-29T11:23:14.400 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-29T11:23:14.411 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-29T11:23:14.411 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-29T11:23:14.500 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-29T11:23:14.501 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-29T11:23:14.797 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-29T11:23:14.958 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 11:21:39 AM UTC. 2024-03-29T11:23:14.966 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:23:15.020 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:23:15.108 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-29T11:23:15.108 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:23:15.108 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-03-29T11:23:15.108 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:23:15.108 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-29T11:23:15.109 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:23:15.109 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-03-29T11:23:15.109 INFO:teuthology.orchestra.run.smithi135.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:23:15.109 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:15.109 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-29T11:23:15.109 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:23:15.109 INFO:teuthology.orchestra.run.smithi135.stdout:Install 2 Packages 2024-03-29T11:23:15.109 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:15.110 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 964 k 2024-03-29T11:23:15.110 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 2.8 M 2024-03-29T11:23:15.110 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-29T11:23:15.243 INFO:teuthology.orchestra.run.smithi135.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 588 kB/s | 79 kB 00:00 2024-03-29T11:23:15.542 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:23:15.622 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:23:15.683 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:23:15.706 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:23:15.852 INFO:teuthology.orchestra.run.smithi135.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 885 kB 00:00 2024-03-29T11:23:15.853 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:15.854 INFO:teuthology.orchestra.run.smithi135.stdout:Total 1.3 MB/s | 964 kB 00:00 2024-03-29T11:23:15.854 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-29T11:23:15.874 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-29T11:23:15.874 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-29T11:23:15.948 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-29T11:23:15.948 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-29T11:23:16.137 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-29T11:23:16.286 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:23:16.360 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:23:16.483 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:23:16.483 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:23:16.483 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:23:16.483 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:23:16.483 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:23:16.483 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:23:16.782 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:23:16.782 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:16.782 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-29T11:23:16.782 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:23:16.782 INFO:teuthology.orchestra.run.smithi194.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:23:16.783 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:23:16.783 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:23:16.783 INFO:teuthology.orchestra.run.smithi194.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:23:16.783 INFO:teuthology.orchestra.run.smithi194.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:23:16.783 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:16.783 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-29T11:23:17.016 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-fuse 2024-03-29T11:23:17.068 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:23:17.068 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:23:17.364 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:23:17.364 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:17.364 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-29T11:23:17.364 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:23:17.364 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:17.364 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-29T11:23:17.576 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 11:21:33 AM UTC. 2024-03-29T11:23:17.643 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-volume 2024-03-29T11:23:17.726 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-29T11:23:17.727 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:23:17.727 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-29T11:23:17.727 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:23:17.727 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-29T11:23:17.727 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:23:17.727 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-29T11:23:17.727 INFO:teuthology.orchestra.run.smithi194.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:23:17.727 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:17.727 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-29T11:23:17.727 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:23:17.728 INFO:teuthology.orchestra.run.smithi194.stdout:Install 2 Packages 2024-03-29T11:23:17.728 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:17.728 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 964 k 2024-03-29T11:23:17.728 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 2.8 M 2024-03-29T11:23:17.728 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-29T11:23:17.852 INFO:teuthology.orchestra.run.smithi194.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 635 kB/s | 79 kB 00:00 2024-03-29T11:23:18.223 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:21:39 AM UTC. 2024-03-29T11:23:18.379 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-29T11:23:18.380 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:23:18.380 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2024-03-29T11:23:18.380 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:23:18.380 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-29T11:23:18.380 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:23:18.380 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2024-03-29T11:23:18.380 INFO:teuthology.orchestra.run.smithi135.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:23:18.380 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:18.380 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-29T11:23:18.380 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:23:18.380 INFO:teuthology.orchestra.run.smithi135.stdout:Install 2 Packages 2024-03-29T11:23:18.380 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:18.380 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 332 k 2024-03-29T11:23:18.381 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 1.5 M 2024-03-29T11:23:18.381 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-29T11:23:18.623 INFO:teuthology.orchestra.run.smithi135.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 283 kB/s | 69 kB 00:00 2024-03-29T11:23:18.661 INFO:teuthology.orchestra.run.smithi194.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 949 kB/s | 885 kB 00:00 2024-03-29T11:23:18.662 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:18.663 INFO:teuthology.orchestra.run.smithi194.stdout:Total 1.0 MB/s | 964 kB 00:00 2024-03-29T11:23:18.663 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-29T11:23:18.687 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-29T11:23:18.687 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-29T11:23:18.765 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-29T11:23:18.766 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-29T11:23:18.949 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-29T11:23:19.049 INFO:teuthology.orchestra.run.smithi135.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 395 kB/s | 264 kB 00:00 2024-03-29T11:23:19.050 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:19.050 INFO:teuthology.orchestra.run.smithi135.stdout:Total 496 kB/s | 332 kB 00:00 2024-03-29T11:23:19.050 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-29T11:23:19.059 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-29T11:23:19.059 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-29T11:23:19.086 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:23:19.098 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-29T11:23:19.098 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-29T11:23:19.152 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:23:19.250 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-29T11:23:19.437 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:23:19.468 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:23:19.484 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:23:19.484 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:23:19.484 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:23:19.485 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:19.888 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:23:19.888 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:23:20.179 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:23:20.179 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:20.179 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-29T11:23:20.180 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:23:20.180 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:20.180 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-29T11:23:20.243 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:23:20.460 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-volume 2024-03-29T11:23:20.565 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:23:20.565 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:20.565 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-29T11:23:20.565 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:23:20.566 INFO:teuthology.orchestra.run.smithi135.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:23:20.566 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:20.566 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-29T11:23:21.002 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install librados-devel 2024-03-29T11:23:21.040 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 11:21:33 AM UTC. 2024-03-29T11:23:21.194 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-29T11:23:21.195 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:23:21.195 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-29T11:23:21.195 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:23:21.195 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-29T11:23:21.195 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:23:21.195 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-29T11:23:21.195 INFO:teuthology.orchestra.run.smithi194.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:23:21.196 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:21.196 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-29T11:23:21.196 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:23:21.196 INFO:teuthology.orchestra.run.smithi194.stdout:Install 2 Packages 2024-03-29T11:23:21.196 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:21.196 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 332 k 2024-03-29T11:23:21.196 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 1.5 M 2024-03-29T11:23:21.196 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-29T11:23:21.325 INFO:teuthology.orchestra.run.smithi194.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 533 kB/s | 69 kB 00:00 2024-03-29T11:23:21.475 INFO:teuthology.orchestra.run.smithi194.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 945 kB/s | 264 kB 00:00 2024-03-29T11:23:21.476 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:21.477 INFO:teuthology.orchestra.run.smithi194.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-29T11:23:21.477 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-29T11:23:21.485 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-29T11:23:21.486 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-29T11:23:21.520 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-29T11:23:21.520 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-29T11:23:21.588 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 11:21:39 AM UTC. 2024-03-29T11:23:21.681 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-29T11:23:21.739 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-29T11:23:21.739 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:23:21.739 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-03-29T11:23:21.739 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:23:21.739 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-29T11:23:21.740 INFO:teuthology.orchestra.run.smithi135.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:23:21.740 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:21.740 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-29T11:23:21.740 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:23:21.740 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-03-29T11:23:21.740 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:21.740 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 127 k 2024-03-29T11:23:21.741 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 456 k 2024-03-29T11:23:21.741 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-29T11:23:21.866 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:23:21.903 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:23:21.919 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:23:21.919 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:23:21.919 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:23:21.919 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:21.946 INFO:teuthology.orchestra.run.smithi135.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 615 kB/s | 127 kB 00:00 2024-03-29T11:23:21.947 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:21.947 INFO:teuthology.orchestra.run.smithi135.stdout:Total 611 kB/s | 127 kB 00:00 2024-03-29T11:23:21.947 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-29T11:23:21.953 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-29T11:23:21.953 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-29T11:23:22.008 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-29T11:23:22.008 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-29T11:23:22.151 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-29T11:23:22.270 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:22.668 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:23:22.742 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:22.993 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:23:22.993 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:22.993 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-29T11:23:22.993 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:23:22.994 INFO:teuthology.orchestra.run.smithi194.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:23:22.994 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:22.994 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-29T11:23:23.040 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:23.040 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:23.040 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-29T11:23:23.040 INFO:teuthology.orchestra.run.smithi135.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:23:23.040 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:23.040 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-29T11:23:23.274 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install libcephfs2 2024-03-29T11:23:23.322 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librados-devel 2024-03-29T11:23:23.852 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 11:21:39 AM UTC. 2024-03-29T11:23:23.902 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 11:21:33 AM UTC. 2024-03-29T11:23:23.955 INFO:teuthology.orchestra.run.smithi135.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:23:24.009 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-29T11:23:24.010 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-03-29T11:23:24.010 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-29T11:23:24.053 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-29T11:23:24.054 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:23:24.054 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-29T11:23:24.054 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:23:24.054 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-29T11:23:24.054 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:23:24.054 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:24.054 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-29T11:23:24.054 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:23:24.055 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-29T11:23:24.055 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:24.055 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 127 k 2024-03-29T11:23:24.055 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 456 k 2024-03-29T11:23:24.055 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-29T11:23:24.095 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install libcephfs-devel 2024-03-29T11:23:24.397 INFO:teuthology.orchestra.run.smithi194.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 370 kB/s | 127 kB 00:00 2024-03-29T11:23:24.398 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:24.398 INFO:teuthology.orchestra.run.smithi194.stdout:Total 369 kB/s | 127 kB 00:00 2024-03-29T11:23:24.398 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-29T11:23:24.403 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-29T11:23:24.403 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-29T11:23:24.459 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-29T11:23:24.459 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-29T11:23:24.601 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-29T11:23:24.675 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 11:21:39 AM UTC. 2024-03-29T11:23:24.703 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:24.840 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-29T11:23:24.857 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:23:24.857 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-03-29T11:23:24.857 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:23:24.858 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-29T11:23:24.858 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:23:24.858 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:24.858 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-29T11:23:24.858 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:23:24.858 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-03-29T11:23:24.858 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:24.858 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 31 k 2024-03-29T11:23:24.858 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 140 k 2024-03-29T11:23:24.858 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-29T11:23:25.092 INFO:teuthology.orchestra.run.smithi135.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 123 kB/s | 31 kB 00:00 2024-03-29T11:23:25.093 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:25.093 INFO:teuthology.orchestra.run.smithi135.stdout:Total 122 kB/s | 31 kB 00:00 2024-03-29T11:23:25.093 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-29T11:23:25.097 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-29T11:23:25.097 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-29T11:23:25.142 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-29T11:23:25.142 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-29T11:23:25.167 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:25.265 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-29T11:23:25.401 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:25.446 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:25.446 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:25.446 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-29T11:23:25.446 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:23:25.446 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:25.446 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-29T11:23:25.656 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install libcephfs2 2024-03-29T11:23:25.792 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:26.055 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:26.055 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:26.055 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-29T11:23:26.055 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:23:26.055 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:26.055 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-29T11:23:26.228 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install librados2 2024-03-29T11:23:26.234 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 11:21:33 AM UTC. 2024-03-29T11:23:26.336 INFO:teuthology.orchestra.run.smithi194.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:23:26.389 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-29T11:23:26.391 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-29T11:23:26.391 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-29T11:23:26.455 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install libcephfs-devel 2024-03-29T11:23:26.795 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 11:21:39 AM UTC. 2024-03-29T11:23:26.892 INFO:teuthology.orchestra.run.smithi135.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:23:26.943 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-29T11:23:26.944 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-03-29T11:23:26.944 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-29T11:23:26.988 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install librbd1 2024-03-29T11:23:27.020 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:54 ago on Fri 29 Mar 2024 11:21:33 AM UTC. 2024-03-29T11:23:27.172 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-29T11:23:27.172 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:23:27.172 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-29T11:23:27.172 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:23:27.172 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-29T11:23:27.173 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:23:27.173 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:27.173 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-29T11:23:27.173 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:23:27.173 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-29T11:23:27.173 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:27.173 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 31 k 2024-03-29T11:23:27.173 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 140 k 2024-03-29T11:23:27.173 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-29T11:23:27.425 INFO:teuthology.orchestra.run.smithi194.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 124 kB/s | 31 kB 00:00 2024-03-29T11:23:27.426 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:27.426 INFO:teuthology.orchestra.run.smithi194.stdout:Total 123 kB/s | 31 kB 00:00 2024-03-29T11:23:27.426 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-29T11:23:27.429 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-29T11:23:27.429 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-29T11:23:27.450 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-29T11:23:27.451 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-29T11:23:27.547 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 11:21:39 AM UTC. 2024-03-29T11:23:27.578 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-29T11:23:27.645 INFO:teuthology.orchestra.run.smithi135.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:23:27.682 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:27.696 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-29T11:23:27.697 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-03-29T11:23:27.697 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-29T11:23:27.750 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-rados 2024-03-29T11:23:28.059 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:28.315 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 11:21:39 AM UTC. 2024-03-29T11:23:28.321 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:28.321 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:28.321 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-29T11:23:28.321 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:23:28.322 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:28.322 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-29T11:23:28.413 INFO:teuthology.orchestra.run.smithi135.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:23:28.465 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-29T11:23:28.466 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-03-29T11:23:28.466 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-29T11:23:28.475 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librados2 2024-03-29T11:23:28.508 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-rgw 2024-03-29T11:23:29.034 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:56 ago on Fri 29 Mar 2024 11:21:33 AM UTC. 2024-03-29T11:23:29.075 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 11:21:39 AM UTC. 2024-03-29T11:23:29.130 INFO:teuthology.orchestra.run.smithi194.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:23:29.176 INFO:teuthology.orchestra.run.smithi135.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:23:29.183 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-29T11:23:29.184 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-29T11:23:29.184 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-29T11:23:29.228 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-29T11:23:29.229 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librbd1 2024-03-29T11:23:29.230 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-03-29T11:23:29.230 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-29T11:23:29.279 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-cephfs 2024-03-29T11:23:29.792 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:56 ago on Fri 29 Mar 2024 11:21:33 AM UTC. 2024-03-29T11:23:29.852 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 11:21:39 AM UTC. 2024-03-29T11:23:29.891 INFO:teuthology.orchestra.run.smithi194.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:23:29.941 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-29T11:23:29.942 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-29T11:23:29.942 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-29T11:23:29.951 INFO:teuthology.orchestra.run.smithi135.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:23:29.992 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rados 2024-03-29T11:23:30.003 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-29T11:23:30.004 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-03-29T11:23:30.004 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-29T11:23:30.054 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-rbd 2024-03-29T11:23:30.559 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:57 ago on Fri 29 Mar 2024 11:21:33 AM UTC. 2024-03-29T11:23:30.624 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:51 ago on Fri 29 Mar 2024 11:21:39 AM UTC. 2024-03-29T11:23:30.657 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:23:30.709 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-29T11:23:30.710 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-29T11:23:30.710 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-29T11:23:30.722 INFO:teuthology.orchestra.run.smithi135.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:23:30.751 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rgw 2024-03-29T11:23:30.774 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-29T11:23:30.775 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2024-03-29T11:23:30.775 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-29T11:23:30.821 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install rbd-fuse 2024-03-29T11:23:31.322 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:58 ago on Fri 29 Mar 2024 11:21:33 AM UTC. 2024-03-29T11:23:31.382 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:52 ago on Fri 29 Mar 2024 11:21:39 AM UTC. 2024-03-29T11:23:31.421 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:23:31.473 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-29T11:23:31.474 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-29T11:23:31.474 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-29T11:23:31.518 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-cephfs 2024-03-29T11:23:31.530 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-29T11:23:31.531 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:23:31.531 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2024-03-29T11:23:31.531 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:23:31.531 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-29T11:23:31.531 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:23:31.532 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:31.532 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-29T11:23:31.532 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:23:31.532 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-03-29T11:23:31.532 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:31.532 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 85 k 2024-03-29T11:23:31.532 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 231 k 2024-03-29T11:23:31.533 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-29T11:23:31.888 INFO:teuthology.orchestra.run.smithi135.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 240 kB/s | 85 kB 00:00 2024-03-29T11:23:31.888 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:31.889 INFO:teuthology.orchestra.run.smithi135.stdout:Total 239 kB/s | 85 kB 00:00 2024-03-29T11:23:31.889 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-29T11:23:31.895 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-29T11:23:31.895 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-29T11:23:31.951 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-29T11:23:31.951 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-29T11:23:32.089 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:59 ago on Fri 29 Mar 2024 11:21:33 AM UTC. 2024-03-29T11:23:32.093 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-29T11:23:32.187 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:23:32.202 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:32.238 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-29T11:23:32.240 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-29T11:23:32.240 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-29T11:23:32.288 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rbd 2024-03-29T11:23:32.622 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:32.856 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:59 ago on Fri 29 Mar 2024 11:21:33 AM UTC. 2024-03-29T11:23:32.907 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:32.907 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:32.907 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-29T11:23:32.907 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:23:32.907 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:32.907 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-29T11:23:32.955 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:23:33.007 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-29T11:23:33.008 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-03-29T11:23:33.008 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-29T11:23:33.054 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-fuse 2024-03-29T11:23:33.164 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install rbd-mirror 2024-03-29T11:23:33.618 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:00 ago on Fri 29 Mar 2024 11:21:33 AM UTC. 2024-03-29T11:23:33.744 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:54 ago on Fri 29 Mar 2024 11:21:39 AM UTC. 2024-03-29T11:23:33.770 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-29T11:23:33.771 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:23:33.771 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-03-29T11:23:33.771 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:23:33.771 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-29T11:23:33.771 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:23:33.771 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:33.771 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-29T11:23:33.771 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:23:33.771 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-29T11:23:33.772 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:33.772 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 85 k 2024-03-29T11:23:33.772 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 231 k 2024-03-29T11:23:33.772 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-29T11:23:33.899 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-29T11:23:33.900 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:23:33.900 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2024-03-29T11:23:33.900 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:23:33.900 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-29T11:23:33.900 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:23:33.900 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:33.900 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-29T11:23:33.900 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:23:33.900 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-03-29T11:23:33.900 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:33.900 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 3.1 M 2024-03-29T11:23:33.900 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 13 M 2024-03-29T11:23:33.900 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-29T11:23:33.978 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 415 kB/s | 85 kB 00:00 2024-03-29T11:23:33.978 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:33.978 INFO:teuthology.orchestra.run.smithi194.stdout:Total 412 kB/s | 85 kB 00:00 2024-03-29T11:23:33.978 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-29T11:23:33.985 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-29T11:23:33.985 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-29T11:23:34.040 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-29T11:23:34.041 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-29T11:23:34.174 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-29T11:23:34.276 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:34.290 INFO:teuthology.orchestra.run.smithi135.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 7.9 MB/s | 3.1 MB 00:00 2024-03-29T11:23:34.291 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:34.291 INFO:teuthology.orchestra.run.smithi135.stdout:Total 7.9 MB/s | 3.1 MB 00:00 2024-03-29T11:23:34.292 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-29T11:23:34.301 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-29T11:23:34.301 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-29T11:23:34.374 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-29T11:23:34.375 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-29T11:23:34.711 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:34.807 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-29T11:23:34.875 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:34.899 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:34.899 INFO:teuthology.orchestra.run.smithi135.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:23:34.899 INFO:teuthology.orchestra.run.smithi135.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:23:34.899 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:23:34.899 INFO:teuthology.orchestra.run.smithi135.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:23:34.899 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:34.998 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:34.999 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:34.999 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-29T11:23:34.999 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:23:34.999 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:34.999 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-29T11:23:35.248 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-mirror 2024-03-29T11:23:35.830 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:02 ago on Fri 29 Mar 2024 11:21:33 AM UTC. 2024-03-29T11:23:35.990 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-29T11:23:35.991 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:23:35.991 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-03-29T11:23:35.991 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:23:35.991 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-29T11:23:35.991 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:23:35.991 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:35.991 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-29T11:23:35.991 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:23:35.991 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-29T11:23:35.991 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:35.992 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 3.1 M 2024-03-29T11:23:35.992 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 13 M 2024-03-29T11:23:35.992 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-29T11:23:36.037 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:36.037 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:36.037 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-29T11:23:36.037 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:23:36.037 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:36.037 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-29T11:23:36.269 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install rbd-nbd 2024-03-29T11:23:36.364 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 8.3 MB/s | 3.1 MB 00:00 2024-03-29T11:23:36.365 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:36.365 INFO:teuthology.orchestra.run.smithi194.stdout:Total 8.3 MB/s | 3.1 MB 00:00 2024-03-29T11:23:36.365 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-29T11:23:36.375 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-29T11:23:36.375 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-29T11:23:36.447 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-29T11:23:36.447 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-29T11:23:36.851 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:57 ago on Fri 29 Mar 2024 11:21:39 AM UTC. 2024-03-29T11:23:36.871 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-29T11:23:36.923 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:36.949 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:36.949 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:23:36.949 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:23:36.949 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:23:36.950 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:23:36.950 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:37.002 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2024-03-29T11:23:37.002 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:23:37.003 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2024-03-29T11:23:37.003 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:23:37.003 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2024-03-29T11:23:37.003 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:23:37.003 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:37.003 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2024-03-29T11:23:37.003 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2024-03-29T11:23:37.003 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2024-03-29T11:23:37.003 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:37.003 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 171 k 2024-03-29T11:23:37.003 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 494 k 2024-03-29T11:23:37.003 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2024-03-29T11:23:37.245 INFO:teuthology.orchestra.run.smithi135.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 711 kB/s | 171 kB 00:00 2024-03-29T11:23:37.246 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:37.246 INFO:teuthology.orchestra.run.smithi135.stdout:Total 707 kB/s | 171 kB 00:00 2024-03-29T11:23:37.246 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2024-03-29T11:23:37.254 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2024-03-29T11:23:37.254 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2024-03-29T11:23:37.309 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2024-03-29T11:23:37.309 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2024-03-29T11:23:37.468 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2024-03-29T11:23:37.577 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:38.035 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:38.094 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:38.094 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:38.094 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-29T11:23:38.094 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:23:38.094 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:38.094 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-29T11:23:38.326 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-nbd 2024-03-29T11:23:38.327 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:38.327 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:38.328 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2024-03-29T11:23:38.328 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:23:38.328 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:38.328 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2024-03-29T11:23:38.582 DEBUG:teuthology.parallel:result is None 2024-03-29T11:23:38.900 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:05 ago on Fri 29 Mar 2024 11:21:33 AM UTC. 2024-03-29T11:23:39.049 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-29T11:23:39.049 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:23:39.049 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-03-29T11:23:39.050 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:23:39.050 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-29T11:23:39.050 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:23:39.050 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:39.050 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-29T11:23:39.050 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-03-29T11:23:39.050 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-03-29T11:23:39.050 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:39.050 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 171 k 2024-03-29T11:23:39.050 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 494 k 2024-03-29T11:23:39.050 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-29T11:23:39.401 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 490 kB/s | 171 kB 00:00 2024-03-29T11:23:39.401 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:39.402 INFO:teuthology.orchestra.run.smithi194.stdout:Total 488 kB/s | 171 kB 00:00 2024-03-29T11:23:39.402 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-29T11:23:39.409 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-29T11:23:39.409 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-29T11:23:39.465 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-29T11:23:39.466 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-29T11:23:39.619 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-29T11:23:39.708 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:40.185 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:40.470 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:23:40.470 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:40.470 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-29T11:23:40.470 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:23:40.470 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:40.470 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-29T11:23:40.686 DEBUG:teuthology.parallel:result is None 2024-03-29T11:23:40.687 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:23:40.836 DEBUG:teuthology.orchestra.run.smithi135:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:23:40.861 INFO:teuthology.orchestra.run.smithi135.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:23:40.861 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:23:40.861 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:23:40.863 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:23:41.074 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:23:41.095 INFO:teuthology.orchestra.run.smithi194.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:23:41.096 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:23:41.096 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:23:41.097 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:23:41.098 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-29T11:23:41.098 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:23:41.131 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-29T11:23:41.131 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:23:41.163 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:23:41.164 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-29T11:23:41.164 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:23:41.202 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:23:41.271 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-29T11:23:41.271 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:23:41.297 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:23:41.364 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:23:41.364 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-29T11:23:41.364 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:23:41.394 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:23:41.463 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-29T11:23:41.463 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:23:41.489 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:23:41.556 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:23:41.556 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-29T11:23:41.556 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:23:41.584 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:23:41.652 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-29T11:23:41.653 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:23:41.679 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:23:41.747 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:23:41.831 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:23:41.831 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 777 /var/log/ceph 2024-03-29T11:23:41.834 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 777 /var/log/ceph 2024-03-29T11:23:41.860 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:23:41.860 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:23:41.902 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:23:41.928 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:23:41.929 DEBUG:teuthology.orchestra.run.smithi135:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:23:41.971 DEBUG:teuthology.orchestra.run.smithi194:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:23:42.007 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:23:42.007 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:23:42.007 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622705', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622705', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi135.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB3FPCTUhwsFHInF9Tv4jgUIcbdIN7FEbfb79V0LlTwdLnCP3Ohx3Nkc0l0n1RnUrjovS7OmvmSzaxp2IxMsJco=', 'smithi194.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCGRy6yfe5LtLCQ8rJvM8g9WvDkJwb+f6+7wdIOVHmvUQjJbOpGy6Cvay+gB1EWaXZ0Us87iEr7S2+5OwPUTSLM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:23:42.008 DEBUG:teuthology.orchestra.run.smithi135:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:23:42.038 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:23:42.053 DEBUG:teuthology.orchestra.run.smithi135:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:23:42.093 DEBUG:teuthology.orchestra.run.smithi194:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:23:42.120 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-29T11:23:42.120 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:23:42.173 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:23:42.173 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_1 2024-03-29T11:23:42.228 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:23:42.228 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:23:42.228 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-29T11:23:42.228 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:23:42.228 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:23:42.229 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-03-29 11:23:35.173754139 +0000 2024-03-29T11:23:42.229 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-03-29 11:20:46.933578390 +0000 2024-03-29T11:23:42.229 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-03-29 11:20:46.933578390 +0000 2024-03-29T11:23:42.229 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-03-29 11:20:46.933578390 +0000 2024-03-29T11:23:42.229 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:23:42.293 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-03-29T11:23:42.294 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-03-29T11:23:42.294 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000254196 s, 2.0 MB/s 2024-03-29T11:23:42.295 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:23:42.350 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_2 2024-03-29T11:23:42.405 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:23:42.405 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:23:42.405 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T11:23:42.405 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:23:42.405 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:23:42.405 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-03-29 11:23:35.173754139 +0000 2024-03-29T11:23:42.405 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-03-29 11:20:47.279570961 +0000 2024-03-29T11:23:42.405 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-03-29 11:20:47.279570961 +0000 2024-03-29T11:23:42.405 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-03-29 11:20:47.279570961 +0000 2024-03-29T11:23:42.405 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:23:42.470 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-03-29T11:23:42.470 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-03-29T11:23:42.470 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.00018849 s, 2.7 MB/s 2024-03-29T11:23:42.471 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:23:42.526 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_3 2024-03-29T11:23:42.581 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:23:42.581 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:23:42.581 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-29T11:23:42.581 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:23:42.581 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:23:42.582 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-03-29 11:23:35.174754117 +0000 2024-03-29T11:23:42.582 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-03-29 11:20:47.613563790 +0000 2024-03-29T11:23:42.582 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-03-29 11:20:47.613563790 +0000 2024-03-29T11:23:42.582 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-03-29 11:20:47.613563790 +0000 2024-03-29T11:23:42.582 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:23:42.646 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-03-29T11:23:42.646 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-03-29T11:23:42.646 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000214312 s, 2.4 MB/s 2024-03-29T11:23:42.648 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:23:42.703 DEBUG:teuthology.orchestra.run.smithi135:> stat /dev/vg_nvme/lv_4 2024-03-29T11:23:42.757 INFO:teuthology.orchestra.run.smithi135.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:23:42.758 INFO:teuthology.orchestra.run.smithi135.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:23:42.758 INFO:teuthology.orchestra.run.smithi135.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-29T11:23:42.758 INFO:teuthology.orchestra.run.smithi135.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:23:42.758 INFO:teuthology.orchestra.run.smithi135.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:23:42.758 INFO:teuthology.orchestra.run.smithi135.stdout:Access: 2024-03-29 11:23:35.174754117 +0000 2024-03-29T11:23:42.758 INFO:teuthology.orchestra.run.smithi135.stdout:Modify: 2024-03-29 11:20:47.953556490 +0000 2024-03-29T11:23:42.758 INFO:teuthology.orchestra.run.smithi135.stdout:Change: 2024-03-29 11:20:47.953556490 +0000 2024-03-29T11:23:42.758 INFO:teuthology.orchestra.run.smithi135.stdout: Birth: 2024-03-29 11:20:47.953556490 +0000 2024-03-29T11:23:42.758 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:23:42.823 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records in 2024-03-29T11:23:42.823 INFO:teuthology.orchestra.run.smithi135.stderr:1+0 records out 2024-03-29T11:23:42.823 INFO:teuthology.orchestra.run.smithi135.stderr:512 bytes copied, 0.000234348 s, 2.2 MB/s 2024-03-29T11:23:42.824 DEBUG:teuthology.orchestra.run.smithi135:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:23:42.879 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:23:42.879 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-29T11:23:42.880 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:23:42.896 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:23:42.896 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_1 2024-03-29T11:23:42.951 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:23:42.952 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:23:42.952 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-29T11:23:42.952 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:23:42.952 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:23:42.952 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-03-29 11:23:37.222763519 +0000 2024-03-29T11:23:42.952 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-03-29 11:20:46.922045350 +0000 2024-03-29T11:23:42.952 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-03-29 11:20:46.922045350 +0000 2024-03-29T11:23:42.952 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-03-29 11:20:46.922045350 +0000 2024-03-29T11:23:42.952 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:23:43.018 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-03-29T11:23:43.019 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-03-29T11:23:43.019 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000264703 s, 1.9 MB/s 2024-03-29T11:23:43.020 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:23:43.076 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_2 2024-03-29T11:23:43.131 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:23:43.131 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:23:43.131 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-29T11:23:43.131 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:23:43.131 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:23:43.132 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-03-29 11:23:37.222763519 +0000 2024-03-29T11:23:43.132 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-03-29 11:20:47.256038519 +0000 2024-03-29T11:23:43.132 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-03-29 11:20:47.256038519 +0000 2024-03-29T11:23:43.132 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-03-29 11:20:47.256038519 +0000 2024-03-29T11:23:43.132 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:23:43.196 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-03-29T11:23:43.196 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-03-29T11:23:43.196 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000206063 s, 2.5 MB/s 2024-03-29T11:23:43.198 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:23:43.253 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_3 2024-03-29T11:23:43.308 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:23:43.308 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:23:43.309 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-29T11:23:43.309 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:23:43.309 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:23:43.309 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-03-29 11:23:37.223763499 +0000 2024-03-29T11:23:43.309 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-03-29 11:20:47.604031402 +0000 2024-03-29T11:23:43.309 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-03-29 11:20:47.604031402 +0000 2024-03-29T11:23:43.309 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-03-29 11:20:47.604031402 +0000 2024-03-29T11:23:43.309 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:23:43.374 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-03-29T11:23:43.374 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-03-29T11:23:43.374 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.00017746 s, 2.9 MB/s 2024-03-29T11:23:43.375 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:23:43.430 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_4 2024-03-29T11:23:43.484 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:23:43.485 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:23:43.485 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-29T11:23:43.485 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:23:43.485 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:23:43.485 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-03-29 11:23:37.223763499 +0000 2024-03-29T11:23:43.485 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-03-29 11:20:47.941024510 +0000 2024-03-29T11:23:43.485 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-03-29 11:20:47.941024510 +0000 2024-03-29T11:23:43.485 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-03-29 11:20:47.941024510 +0000 2024-03-29T11:23:43.485 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:23:43.550 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-03-29T11:23:43.550 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-03-29T11:23:43.550 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000255128 s, 2.0 MB/s 2024-03-29T11:23:43.551 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:23:43.607 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:23:43.607 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi135.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi194.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:23:43.607 INFO:tasks.ceph:Generating config... 2024-03-29T11:23:43.608 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:23:43.608 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:23:43.609 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:23:43.609 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:23:43.609 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:23:43.609 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:23:43.609 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:23:43.609 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:23:43.609 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:23:43.609 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:23:43.609 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:23:43.609 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:23:43.609 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:23:43.609 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:23:43.609 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:23:43.609 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:23:43.610 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:23:43.610 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:23:43.610 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:23:43.610 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:23:43.610 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:23:43.610 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:23:43.610 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:23:43.610 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-29T11:23:43.610 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:23:43.610 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:23:43.610 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:23:43.610 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:23:43.610 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:23:43.610 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:23:43.611 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:23:43.611 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:23:43.611 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:23:43.611 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:23:43.611 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:23:43.611 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:23:43.611 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:23:43.611 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:23:43.611 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:23:43.611 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:23:43.611 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:23:43.611 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:23:43.611 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:23:43.611 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:23:43.611 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:23:43.612 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:23:43.612 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:23:43.612 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:23:43.612 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:23:43.612 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:23:43.672 INFO:teuthology.orchestra.run.smithi135.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:23:43.675 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:23:43.721 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:23:43.749 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.135:6789]'), ('mon.c', '[v1:172.21.15.135:6790]'), ('mon.b', '[v1:172.21.15.194:6789]')] 2024-03-29T11:23:43.750 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.135:6789],[v1:172.21.15.135:6790],[v1:172.21.15.194:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:23:43.750 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-29T11:23:43.750 DEBUG:teuthology.orchestra.run.smithi135:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:23:43.804 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.135:6789]' --addv c '[v1:172.21.15.135:6790]' --addv b '[v1:172.21.15.194:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:23:43.888 INFO:teuthology.orchestra.run.smithi135.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:23:43.888 INFO:teuthology.orchestra.run.smithi135.stdout:monmaptool: generated fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:23:43.888 INFO:teuthology.orchestra.run.smithi135.stdout:setting min_mon_release = quincy 2024-03-29T11:23:43.888 INFO:teuthology.orchestra.run.smithi135.stdout:epoch 0 2024-03-29T11:23:43.889 INFO:teuthology.orchestra.run.smithi135.stdout:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:23:43.889 INFO:teuthology.orchestra.run.smithi135.stdout:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:23:43.889 INFO:teuthology.orchestra.run.smithi135.stdout:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:23:43.889 INFO:teuthology.orchestra.run.smithi135.stdout:min_mon_release 17 (quincy) 2024-03-29T11:23:43.889 INFO:teuthology.orchestra.run.smithi135.stdout:election_strategy: 1 2024-03-29T11:23:43.889 INFO:teuthology.orchestra.run.smithi135.stdout:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:23:43.889 INFO:teuthology.orchestra.run.smithi135.stdout:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:23:43.889 INFO:teuthology.orchestra.run.smithi135.stdout:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:23:43.889 INFO:teuthology.orchestra.run.smithi135.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:23:43.890 DEBUG:teuthology.orchestra.run.smithi135:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:23:43.904 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 8b499097-26c8-45be-955e-1a34503f2b8b... 2024-03-29T11:23:43.905 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:23:43.947 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:23:43.994 INFO:teuthology.orchestra.run.smithi135.stdout:[global] 2024-03-29T11:23:43.994 INFO:teuthology.orchestra.run.smithi135.stdout: chdir = "" 2024-03-29T11:23:43.994 INFO:teuthology.orchestra.run.smithi135.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:23:43.994 INFO:teuthology.orchestra.run.smithi135.stdout: auth supported = cephx 2024-03-29T11:23:43.995 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:43.995 INFO:teuthology.orchestra.run.smithi135.stdout: filestore xattr use omap = true 2024-03-29T11:23:43.995 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:43.995 INFO:teuthology.orchestra.run.smithi135.stdout: mon clock drift allowed = 1.000 2024-03-29T11:23:43.995 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:43.995 INFO:teuthology.orchestra.run.smithi135.stdout: osd crush chooseleaf type = 0 2024-03-29T11:23:43.995 INFO:teuthology.orchestra.run.smithi135.stdout: auth debug = true 2024-03-29T11:23:43.995 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:43.995 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on old message = true 2024-03-29T11:23:43.995 INFO:teuthology.orchestra.run.smithi135.stdout: ms die on bug = true 2024-03-29T11:23:43.995 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:43.996 INFO:teuthology.orchestra.run.smithi135.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:23:43.996 INFO:teuthology.orchestra.run.smithi135.stdout: mon pg warn max object skew = 0 2024-03-29T11:23:43.996 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:43.996 INFO:teuthology.orchestra.run.smithi135.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:23:43.996 INFO:teuthology.orchestra.run.smithi135.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:23:43.996 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:43.996 INFO:teuthology.orchestra.run.smithi135.stdout: osd pool default size = 2 2024-03-29T11:23:43.996 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:43.996 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd allow primary affinity = true 2024-03-29T11:23:43.996 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd allow pg remap = true 2024-03-29T11:23:43.996 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:23:43.996 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:23:43.996 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on no sortbitwise = false 2024-03-29T11:23:43.996 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:23:43.997 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on too few osds = false 2024-03-29T11:23:43.997 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:23:43.997 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:23:43.997 INFO:teuthology.orchestra.run.smithi135.stdout: mon_allow_pool_size_one = true 2024-03-29T11:23:43.997 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:43.997 INFO:teuthology.orchestra.run.smithi135.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:23:43.997 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:43.997 INFO:teuthology.orchestra.run.smithi135.stdout: osd default data pool replay window = 5 2024-03-29T11:23:43.997 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:43.997 INFO:teuthology.orchestra.run.smithi135.stdout: mon allow pool delete = true 2024-03-29T11:23:43.997 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:43.997 INFO:teuthology.orchestra.run.smithi135.stdout: mon cluster log file level = debug 2024-03-29T11:23:43.997 INFO:teuthology.orchestra.run.smithi135.stdout: debug asserts on shutdown = true 2024-03-29T11:23:43.997 INFO:teuthology.orchestra.run.smithi135.stdout: mon health detail to clog = false 2024-03-29T11:23:43.997 INFO:teuthology.orchestra.run.smithi135.stdout: mon host = "[v1:172.21.15.135:6789],[v1:172.21.15.135:6790],[v1:172.21.15.194:6789]" 2024-03-29T11:23:43.998 INFO:teuthology.orchestra.run.smithi135.stdout: lockdep = True 2024-03-29T11:23:43.998 INFO:teuthology.orchestra.run.smithi135.stdout: mon client directed command retry = 5 2024-03-29T11:23:43.998 INFO:teuthology.orchestra.run.smithi135.stdout: mon election default strategy = 1 2024-03-29T11:23:43.998 INFO:teuthology.orchestra.run.smithi135.stdout: ms bind msgr2 = False 2024-03-29T11:23:43.998 INFO:teuthology.orchestra.run.smithi135.stdout: ms inject socket failures = 5000 2024-03-29T11:23:43.998 INFO:teuthology.orchestra.run.smithi135.stdout: ms type = async 2024-03-29T11:23:43.998 INFO:teuthology.orchestra.run.smithi135.stdout: fsid = 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:23:43.998 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:43.998 INFO:teuthology.orchestra.run.smithi135.stdout:[osd] 2024-03-29T11:23:43.998 INFO:teuthology.orchestra.run.smithi135.stdout: osd journal size = 100 2024-03-29T11:23:43.998 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:43.998 INFO:teuthology.orchestra.run.smithi135.stdout: osd scrub load threshold = 5.0 2024-03-29T11:23:43.998 INFO:teuthology.orchestra.run.smithi135.stdout: osd scrub max interval = 600 2024-03-29T11:23:43.999 INFO:teuthology.orchestra.run.smithi135.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:23:43.999 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:43.999 INFO:teuthology.orchestra.run.smithi135.stdout: osd recover clone overlap = true 2024-03-29T11:23:43.999 INFO:teuthology.orchestra.run.smithi135.stdout: osd recovery max chunk = 1048576 2024-03-29T11:23:43.999 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:43.999 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug shutdown = true 2024-03-29T11:23:43.999 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug op order = true 2024-03-29T11:23:43.999 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug verify stray on activate = true 2024-03-29T11:23:43.999 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug trim objects = true 2024-03-29T11:23:43.999 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:43.999 INFO:teuthology.orchestra.run.smithi135.stdout: osd open classes on start = true 2024-03-29T11:23:43.999 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug pg log writeout = true 2024-03-29T11:23:43.999 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:43.999 INFO:teuthology.orchestra.run.smithi135.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:23:43.999 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:44.000 INFO:teuthology.orchestra.run.smithi135.stdout: osd map max advance = 10 2024-03-29T11:23:44.000 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:44.000 INFO:teuthology.orchestra.run.smithi135.stdout: journal zero on create = true 2024-03-29T11:23:44.000 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:44.000 INFO:teuthology.orchestra.run.smithi135.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:23:44.000 INFO:teuthology.orchestra.run.smithi135.stdout: filestore apply finisher threads = 3 2024-03-29T11:23:44.000 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:44.000 INFO:teuthology.orchestra.run.smithi135.stdout: bdev debug aio = true 2024-03-29T11:23:44.000 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug misdirected ops = true 2024-03-29T11:23:44.000 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore block size = 96636764160 2024-03-29T11:23:44.000 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore compression algorithm = snappy 2024-03-29T11:23:44.000 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore compression mode = aggressive 2024-03-29T11:23:44.000 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore fsck on mount = True 2024-03-29T11:23:44.000 INFO:teuthology.orchestra.run.smithi135.stdout: bluestore zero block detection = True 2024-03-29T11:23:44.000 INFO:teuthology.orchestra.run.smithi135.stdout: debug bluefs = 20 2024-03-29T11:23:44.000 INFO:teuthology.orchestra.run.smithi135.stdout: debug bluestore = 20 2024-03-29T11:23:44.001 INFO:teuthology.orchestra.run.smithi135.stdout: debug monc = 20 2024-03-29T11:23:44.001 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-29T11:23:44.001 INFO:teuthology.orchestra.run.smithi135.stdout: debug osd = 20 2024-03-29T11:23:44.001 INFO:teuthology.orchestra.run.smithi135.stdout: debug rocksdb = 10 2024-03-29T11:23:44.001 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:23:44.001 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd full ratio = 0.9 2024-03-29T11:23:44.001 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:23:44.001 INFO:teuthology.orchestra.run.smithi135.stdout: osd class default list = * 2024-03-29T11:23:44.001 INFO:teuthology.orchestra.run.smithi135.stdout: osd class load list = * 2024-03-29T11:23:44.001 INFO:teuthology.orchestra.run.smithi135.stdout: osd client watch timeout = 120 2024-03-29T11:23:44.001 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug verify cached snaps = True 2024-03-29T11:23:44.001 INFO:teuthology.orchestra.run.smithi135.stdout: osd debug verify missing on start = True 2024-03-29T11:23:44.001 INFO:teuthology.orchestra.run.smithi135.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:23:44.001 INFO:teuthology.orchestra.run.smithi135.stdout: osd mclock override recovery settings = True 2024-03-29T11:23:44.002 INFO:teuthology.orchestra.run.smithi135.stdout: osd objectstore = bluestore 2024-03-29T11:23:44.002 INFO:teuthology.orchestra.run.smithi135.stdout: osd op queue = debug_random 2024-03-29T11:23:44.002 INFO:teuthology.orchestra.run.smithi135.stdout: osd op queue cut off = debug_random 2024-03-29T11:23:44.002 INFO:teuthology.orchestra.run.smithi135.stdout: osd shutdown pgref assert = True 2024-03-29T11:23:44.002 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:44.002 INFO:teuthology.orchestra.run.smithi135.stdout:[mgr] 2024-03-29T11:23:44.002 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-29T11:23:44.002 INFO:teuthology.orchestra.run.smithi135.stdout: debug mgr = 20 2024-03-29T11:23:44.002 INFO:teuthology.orchestra.run.smithi135.stdout: debug mon = 20 2024-03-29T11:23:44.002 INFO:teuthology.orchestra.run.smithi135.stdout: debug auth = 20 2024-03-29T11:23:44.002 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:23:44.002 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:23:44.003 INFO:teuthology.orchestra.run.smithi135.stdout: mgr/telemetry/nag = false 2024-03-29T11:23:44.003 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:44.003 INFO:teuthology.orchestra.run.smithi135.stdout:[mon] 2024-03-29T11:23:44.003 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-29T11:23:44.003 INFO:teuthology.orchestra.run.smithi135.stdout: debug mon = 20 2024-03-29T11:23:44.003 INFO:teuthology.orchestra.run.smithi135.stdout: debug paxos = 20 2024-03-29T11:23:44.003 INFO:teuthology.orchestra.run.smithi135.stdout: debug auth = 20 2024-03-29T11:23:44.003 INFO:teuthology.orchestra.run.smithi135.stdout: mon data avail warn = 5 2024-03-29T11:23:44.003 INFO:teuthology.orchestra.run.smithi135.stdout: mon mgr mkfs grace = 240 2024-03-29T11:23:44.003 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:23:44.003 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd reporter subtree level = osd 2024-03-29T11:23:44.003 INFO:teuthology.orchestra.run.smithi135.stdout: mon osd prime pg temp = true 2024-03-29T11:23:44.003 INFO:teuthology.orchestra.run.smithi135.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:23:44.003 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:44.003 INFO:teuthology.orchestra.run.smithi135.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:23:44.004 INFO:teuthology.orchestra.run.smithi135.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:23:44.004 INFO:teuthology.orchestra.run.smithi135.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:23:44.004 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:44.004 INFO:teuthology.orchestra.run.smithi135.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:23:44.004 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:23:44.004 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:23:44.004 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:44.004 INFO:teuthology.orchestra.run.smithi135.stdout: # 1m isn't quite enough 2024-03-29T11:23:44.004 INFO:teuthology.orchestra.run.smithi135.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:23:44.004 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:44.004 INFO:teuthology.orchestra.run.smithi135.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:23:44.004 INFO:teuthology.orchestra.run.smithi135.stdout: mon min osdmap epochs = 50 2024-03-29T11:23:44.004 INFO:teuthology.orchestra.run.smithi135.stdout: mon osdmap full prune interval = 2 2024-03-29T11:23:44.004 INFO:teuthology.orchestra.run.smithi135.stdout: mon osdmap full prune min = 15 2024-03-29T11:23:44.004 INFO:teuthology.orchestra.run.smithi135.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:23:44.005 INFO:teuthology.orchestra.run.smithi135.stdout: mon scrub interval = 300 2024-03-29T11:23:44.005 INFO:teuthology.orchestra.run.smithi135.stdout: mon warn on pool no app = False 2024-03-29T11:23:44.005 INFO:teuthology.orchestra.run.smithi135.stdout: paxos service trim min = 10 2024-03-29T11:23:44.005 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:23:44.005 INFO:teuthology.orchestra.run.smithi135.stdout:[client] 2024-03-29T11:23:44.005 INFO:teuthology.orchestra.run.smithi135.stdout: rgw cache enabled = true 2024-03-29T11:23:44.005 INFO:teuthology.orchestra.run.smithi135.stdout: rgw enable ops log = true 2024-03-29T11:23:44.005 INFO:teuthology.orchestra.run.smithi135.stdout: rgw enable usage log = true 2024-03-29T11:23:44.005 INFO:teuthology.orchestra.run.smithi135.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:23:44.005 INFO:teuthology.orchestra.run.smithi135.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:23:44.005 INFO:teuthology.orchestra.run.smithi135.stdout: debug monc = 20 2024-03-29T11:23:44.005 INFO:teuthology.orchestra.run.smithi135.stdout: debug ms = 1 2024-03-29T11:23:44.005 INFO:teuthology.orchestra.run.smithi135.stdout: debug objecter = 20 2024-03-29T11:23:44.005 INFO:teuthology.orchestra.run.smithi135.stdout: debug rados = 20 2024-03-29T11:23:44.005 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.a] 2024-03-29T11:23:44.005 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.c] 2024-03-29T11:23:44.006 INFO:teuthology.orchestra.run.smithi135.stdout:[mon.b] 2024-03-29T11:23:44.006 INFO:teuthology.orchestra.run.smithi194.stdout:[global] 2024-03-29T11:23:44.006 INFO:teuthology.orchestra.run.smithi194.stdout: chdir = "" 2024-03-29T11:23:44.006 INFO:teuthology.orchestra.run.smithi194.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:23:44.006 INFO:teuthology.orchestra.run.smithi194.stdout: auth supported = cephx 2024-03-29T11:23:44.006 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.006 INFO:teuthology.orchestra.run.smithi194.stdout: filestore xattr use omap = true 2024-03-29T11:23:44.006 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.006 INFO:teuthology.orchestra.run.smithi194.stdout: mon clock drift allowed = 1.000 2024-03-29T11:23:44.007 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.007 INFO:teuthology.orchestra.run.smithi194.stdout: osd crush chooseleaf type = 0 2024-03-29T11:23:44.007 INFO:teuthology.orchestra.run.smithi194.stdout: auth debug = true 2024-03-29T11:23:44.007 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.007 INFO:teuthology.orchestra.run.smithi194.stdout: ms die on old message = true 2024-03-29T11:23:44.007 INFO:teuthology.orchestra.run.smithi194.stdout: ms die on bug = true 2024-03-29T11:23:44.007 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.007 INFO:teuthology.orchestra.run.smithi194.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:23:44.007 INFO:teuthology.orchestra.run.smithi194.stdout: mon pg warn max object skew = 0 2024-03-29T11:23:44.007 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.007 INFO:teuthology.orchestra.run.smithi194.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:23:44.007 INFO:teuthology.orchestra.run.smithi194.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:23:44.007 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.007 INFO:teuthology.orchestra.run.smithi194.stdout: osd pool default size = 2 2024-03-29T11:23:44.008 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.008 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd allow primary affinity = true 2024-03-29T11:23:44.008 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd allow pg remap = true 2024-03-29T11:23:44.008 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:23:44.008 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:23:44.008 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on no sortbitwise = false 2024-03-29T11:23:44.008 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:23:44.008 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on too few osds = false 2024-03-29T11:23:44.008 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:23:44.008 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:23:44.008 INFO:teuthology.orchestra.run.smithi194.stdout: mon_allow_pool_size_one = true 2024-03-29T11:23:44.008 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.008 INFO:teuthology.orchestra.run.smithi194.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:23:44.008 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.008 INFO:teuthology.orchestra.run.smithi194.stdout: osd default data pool replay window = 5 2024-03-29T11:23:44.008 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.009 INFO:teuthology.orchestra.run.smithi194.stdout: mon allow pool delete = true 2024-03-29T11:23:44.009 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.009 INFO:teuthology.orchestra.run.smithi194.stdout: mon cluster log file level = debug 2024-03-29T11:23:44.009 INFO:teuthology.orchestra.run.smithi194.stdout: debug asserts on shutdown = true 2024-03-29T11:23:44.009 INFO:teuthology.orchestra.run.smithi194.stdout: mon health detail to clog = false 2024-03-29T11:23:44.009 INFO:teuthology.orchestra.run.smithi194.stdout: mon host = "[v1:172.21.15.135:6789],[v1:172.21.15.135:6790],[v1:172.21.15.194:6789]" 2024-03-29T11:23:44.009 INFO:teuthology.orchestra.run.smithi194.stdout: lockdep = True 2024-03-29T11:23:44.009 INFO:teuthology.orchestra.run.smithi194.stdout: mon client directed command retry = 5 2024-03-29T11:23:44.009 INFO:teuthology.orchestra.run.smithi194.stdout: mon election default strategy = 1 2024-03-29T11:23:44.009 INFO:teuthology.orchestra.run.smithi194.stdout: ms bind msgr2 = False 2024-03-29T11:23:44.009 INFO:teuthology.orchestra.run.smithi194.stdout: ms inject socket failures = 5000 2024-03-29T11:23:44.009 INFO:teuthology.orchestra.run.smithi194.stdout: ms type = async 2024-03-29T11:23:44.009 INFO:teuthology.orchestra.run.smithi194.stdout: fsid = 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:23:44.009 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.009 INFO:teuthology.orchestra.run.smithi194.stdout:[osd] 2024-03-29T11:23:44.010 INFO:teuthology.orchestra.run.smithi194.stdout: osd journal size = 100 2024-03-29T11:23:44.010 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.010 INFO:teuthology.orchestra.run.smithi194.stdout: osd scrub load threshold = 5.0 2024-03-29T11:23:44.010 INFO:teuthology.orchestra.run.smithi194.stdout: osd scrub max interval = 600 2024-03-29T11:23:44.010 INFO:teuthology.orchestra.run.smithi194.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:23:44.010 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.010 INFO:teuthology.orchestra.run.smithi194.stdout: osd recover clone overlap = true 2024-03-29T11:23:44.010 INFO:teuthology.orchestra.run.smithi194.stdout: osd recovery max chunk = 1048576 2024-03-29T11:23:44.010 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.010 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug shutdown = true 2024-03-29T11:23:44.010 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug op order = true 2024-03-29T11:23:44.011 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug verify stray on activate = true 2024-03-29T11:23:44.011 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug trim objects = true 2024-03-29T11:23:44.011 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.011 INFO:teuthology.orchestra.run.smithi194.stdout: osd open classes on start = true 2024-03-29T11:23:44.011 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug pg log writeout = true 2024-03-29T11:23:44.011 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.011 INFO:teuthology.orchestra.run.smithi194.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:23:44.011 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.011 INFO:teuthology.orchestra.run.smithi194.stdout: osd map max advance = 10 2024-03-29T11:23:44.011 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.011 INFO:teuthology.orchestra.run.smithi194.stdout: journal zero on create = true 2024-03-29T11:23:44.011 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.011 INFO:teuthology.orchestra.run.smithi194.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:23:44.011 INFO:teuthology.orchestra.run.smithi194.stdout: filestore apply finisher threads = 3 2024-03-29T11:23:44.012 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.012 INFO:teuthology.orchestra.run.smithi194.stdout: bdev debug aio = true 2024-03-29T11:23:44.012 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug misdirected ops = true 2024-03-29T11:23:44.012 INFO:teuthology.orchestra.run.smithi194.stdout: bluestore block size = 96636764160 2024-03-29T11:23:44.012 INFO:teuthology.orchestra.run.smithi194.stdout: bluestore compression algorithm = snappy 2024-03-29T11:23:44.012 INFO:teuthology.orchestra.run.smithi194.stdout: bluestore compression mode = aggressive 2024-03-29T11:23:44.012 INFO:teuthology.orchestra.run.smithi194.stdout: bluestore fsck on mount = True 2024-03-29T11:23:44.012 INFO:teuthology.orchestra.run.smithi194.stdout: bluestore zero block detection = True 2024-03-29T11:23:44.012 INFO:teuthology.orchestra.run.smithi194.stdout: debug bluefs = 20 2024-03-29T11:23:44.012 INFO:teuthology.orchestra.run.smithi194.stdout: debug bluestore = 20 2024-03-29T11:23:44.012 INFO:teuthology.orchestra.run.smithi194.stdout: debug monc = 20 2024-03-29T11:23:44.012 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 1 2024-03-29T11:23:44.012 INFO:teuthology.orchestra.run.smithi194.stdout: debug osd = 20 2024-03-29T11:23:44.012 INFO:teuthology.orchestra.run.smithi194.stdout: debug rocksdb = 10 2024-03-29T11:23:44.012 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:23:44.013 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd full ratio = 0.9 2024-03-29T11:23:44.013 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:23:44.013 INFO:teuthology.orchestra.run.smithi194.stdout: osd class default list = * 2024-03-29T11:23:44.013 INFO:teuthology.orchestra.run.smithi194.stdout: osd class load list = * 2024-03-29T11:23:44.013 INFO:teuthology.orchestra.run.smithi194.stdout: osd client watch timeout = 120 2024-03-29T11:23:44.013 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug verify cached snaps = True 2024-03-29T11:23:44.013 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug verify missing on start = True 2024-03-29T11:23:44.013 INFO:teuthology.orchestra.run.smithi194.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:23:44.013 INFO:teuthology.orchestra.run.smithi194.stdout: osd mclock override recovery settings = True 2024-03-29T11:23:44.013 INFO:teuthology.orchestra.run.smithi194.stdout: osd objectstore = bluestore 2024-03-29T11:23:44.013 INFO:teuthology.orchestra.run.smithi194.stdout: osd op queue = debug_random 2024-03-29T11:23:44.013 INFO:teuthology.orchestra.run.smithi194.stdout: osd op queue cut off = debug_random 2024-03-29T11:23:44.013 INFO:teuthology.orchestra.run.smithi194.stdout: osd shutdown pgref assert = True 2024-03-29T11:23:44.013 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.013 INFO:teuthology.orchestra.run.smithi194.stdout:[mgr] 2024-03-29T11:23:44.014 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 1 2024-03-29T11:23:44.014 INFO:teuthology.orchestra.run.smithi194.stdout: debug mgr = 20 2024-03-29T11:23:44.014 INFO:teuthology.orchestra.run.smithi194.stdout: debug mon = 20 2024-03-29T11:23:44.014 INFO:teuthology.orchestra.run.smithi194.stdout: debug auth = 20 2024-03-29T11:23:44.014 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:23:44.014 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:23:44.014 INFO:teuthology.orchestra.run.smithi194.stdout: mgr/telemetry/nag = false 2024-03-29T11:23:44.014 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.014 INFO:teuthology.orchestra.run.smithi194.stdout:[mon] 2024-03-29T11:23:44.014 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 1 2024-03-29T11:23:44.014 INFO:teuthology.orchestra.run.smithi194.stdout: debug mon = 20 2024-03-29T11:23:44.014 INFO:teuthology.orchestra.run.smithi194.stdout: debug paxos = 20 2024-03-29T11:23:44.014 INFO:teuthology.orchestra.run.smithi194.stdout: debug auth = 20 2024-03-29T11:23:44.014 INFO:teuthology.orchestra.run.smithi194.stdout: mon data avail warn = 5 2024-03-29T11:23:44.015 INFO:teuthology.orchestra.run.smithi194.stdout: mon mgr mkfs grace = 240 2024-03-29T11:23:44.015 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:23:44.015 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd reporter subtree level = osd 2024-03-29T11:23:44.015 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd prime pg temp = true 2024-03-29T11:23:44.015 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:23:44.015 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.015 INFO:teuthology.orchestra.run.smithi194.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:23:44.015 INFO:teuthology.orchestra.run.smithi194.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:23:44.015 INFO:teuthology.orchestra.run.smithi194.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:23:44.015 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.015 INFO:teuthology.orchestra.run.smithi194.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:23:44.015 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:23:44.015 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:23:44.015 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.015 INFO:teuthology.orchestra.run.smithi194.stdout: # 1m isn't quite enough 2024-03-29T11:23:44.016 INFO:teuthology.orchestra.run.smithi194.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:23:44.016 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.016 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:23:44.016 INFO:teuthology.orchestra.run.smithi194.stdout: mon min osdmap epochs = 50 2024-03-29T11:23:44.016 INFO:teuthology.orchestra.run.smithi194.stdout: mon osdmap full prune interval = 2 2024-03-29T11:23:44.016 INFO:teuthology.orchestra.run.smithi194.stdout: mon osdmap full prune min = 15 2024-03-29T11:23:44.016 INFO:teuthology.orchestra.run.smithi194.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:23:44.016 INFO:teuthology.orchestra.run.smithi194.stdout: mon scrub interval = 300 2024-03-29T11:23:44.016 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on pool no app = False 2024-03-29T11:23:44.016 INFO:teuthology.orchestra.run.smithi194.stdout: paxos service trim min = 10 2024-03-29T11:23:44.016 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-29T11:23:44.016 INFO:teuthology.orchestra.run.smithi194.stdout:[client] 2024-03-29T11:23:44.016 INFO:teuthology.orchestra.run.smithi194.stdout: rgw cache enabled = true 2024-03-29T11:23:44.017 INFO:teuthology.orchestra.run.smithi194.stdout: rgw enable ops log = true 2024-03-29T11:23:44.017 INFO:teuthology.orchestra.run.smithi194.stdout: rgw enable usage log = true 2024-03-29T11:23:44.017 INFO:teuthology.orchestra.run.smithi194.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:23:44.017 INFO:teuthology.orchestra.run.smithi194.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:23:44.017 INFO:teuthology.orchestra.run.smithi194.stdout: debug monc = 20 2024-03-29T11:23:44.017 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 1 2024-03-29T11:23:44.017 INFO:teuthology.orchestra.run.smithi194.stdout: debug objecter = 20 2024-03-29T11:23:44.017 INFO:teuthology.orchestra.run.smithi194.stdout: debug rados = 20 2024-03-29T11:23:44.017 INFO:teuthology.orchestra.run.smithi194.stdout:[mon.a] 2024-03-29T11:23:44.017 INFO:teuthology.orchestra.run.smithi194.stdout:[mon.c] 2024-03-29T11:23:44.017 INFO:teuthology.orchestra.run.smithi194.stdout:[mon.b] 2024-03-29T11:23:44.018 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:23:44.019 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:23:44.073 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:23:44.073 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-29T11:23:44.074 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:23:44.088 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-29T11:23:44.088 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:23:44.142 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi135.front.sepia.ceph.com 2024-03-29T11:23:44.143 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-29T11:23:44.143 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:23:44.143 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:23:44.222 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-29T11:23:44.222 DEBUG:teuthology.orchestra.run.smithi135:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:23:44.277 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi194.front.sepia.ceph.com 2024-03-29T11:23:44.277 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-29T11:23:44.277 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:23:44.277 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:23:44.317 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-29T11:23:44.318 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:23:44.373 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:23:44.373 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:23:44.373 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:23:44.428 INFO:teuthology.orchestra.run.smithi135.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:23:44.431 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:23:44.498 INFO:teuthology.orchestra.run.smithi194.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:23:44.500 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:23:44.500 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:23:44.500 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:23:44.543 INFO:teuthology.orchestra.run.smithi135.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:23:44.558 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:23:44.600 INFO:teuthology.orchestra.run.smithi194.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:23:44.615 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:23:44.616 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi135.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi194.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:23:44.616 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:23:44.642 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:23:44.642 INFO:tasks.ceph:role: osd.0 2024-03-29T11:23:44.642 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi135.front.sepia.ceph.com 2024-03-29T11:23:44.642 DEBUG:teuthology.orchestra.run.smithi135:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:23:44.716 INFO:teuthology.orchestra.run.smithi135.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:23:44.716 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:23:44.716 INFO:teuthology.orchestra.run.smithi135.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:23:44.716 INFO:teuthology.orchestra.run.smithi135.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:23:44.716 INFO:teuthology.orchestra.run.smithi135.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:23:44.717 INFO:teuthology.orchestra.run.smithi135.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:23:44.717 INFO:teuthology.orchestra.run.smithi135.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:23:44.717 INFO:teuthology.orchestra.run.smithi135.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:23:44.717 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:23:44.717 INFO:teuthology.orchestra.run.smithi135.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:23:45.179 INFO:teuthology.orchestra.run.smithi135.stdout:Discarding blocks...Done. 2024-03-29T11:23:45.181 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi135.front.sepia.ceph.com -o noatime 2024-03-29T11:23:45.181 DEBUG:teuthology.orchestra.run.smithi135:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:23:45.312 DEBUG:teuthology.orchestra.run.smithi135:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:23:45.346 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:23:45.415 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:23:45.415 INFO:tasks.ceph:role: osd.1 2024-03-29T11:23:45.415 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi135.front.sepia.ceph.com 2024-03-29T11:23:45.415 DEBUG:teuthology.orchestra.run.smithi135:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:23:45.489 INFO:teuthology.orchestra.run.smithi135.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:23:45.489 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:23:45.489 INFO:teuthology.orchestra.run.smithi135.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:23:45.489 INFO:teuthology.orchestra.run.smithi135.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:23:45.489 INFO:teuthology.orchestra.run.smithi135.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:23:45.489 INFO:teuthology.orchestra.run.smithi135.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:23:45.490 INFO:teuthology.orchestra.run.smithi135.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:23:45.490 INFO:teuthology.orchestra.run.smithi135.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:23:45.490 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:23:45.490 INFO:teuthology.orchestra.run.smithi135.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:23:45.932 INFO:teuthology.orchestra.run.smithi135.stdout:Discarding blocks...Done. 2024-03-29T11:23:45.933 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi135.front.sepia.ceph.com -o noatime 2024-03-29T11:23:45.934 DEBUG:teuthology.orchestra.run.smithi135:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:23:46.010 DEBUG:teuthology.orchestra.run.smithi135:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:23:46.081 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:23:46.148 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:23:46.149 INFO:tasks.ceph:role: osd.2 2024-03-29T11:23:46.149 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi135.front.sepia.ceph.com 2024-03-29T11:23:46.149 DEBUG:teuthology.orchestra.run.smithi135:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:23:46.223 INFO:teuthology.orchestra.run.smithi135.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:23:46.223 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:23:46.223 INFO:teuthology.orchestra.run.smithi135.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:23:46.223 INFO:teuthology.orchestra.run.smithi135.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:23:46.223 INFO:teuthology.orchestra.run.smithi135.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:23:46.223 INFO:teuthology.orchestra.run.smithi135.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:23:46.223 INFO:teuthology.orchestra.run.smithi135.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:23:46.223 INFO:teuthology.orchestra.run.smithi135.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:23:46.223 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:23:46.223 INFO:teuthology.orchestra.run.smithi135.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:23:46.706 INFO:teuthology.orchestra.run.smithi135.stdout:Discarding blocks...Done. 2024-03-29T11:23:46.708 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi135.front.sepia.ceph.com -o noatime 2024-03-29T11:23:46.708 DEBUG:teuthology.orchestra.run.smithi135:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:23:46.805 DEBUG:teuthology.orchestra.run.smithi135:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:23:46.836 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:23:46.903 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:23:46.903 INFO:tasks.ceph:role: osd.3 2024-03-29T11:23:46.903 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi135.front.sepia.ceph.com 2024-03-29T11:23:46.903 DEBUG:teuthology.orchestra.run.smithi135:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:23:46.978 INFO:teuthology.orchestra.run.smithi135.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:23:46.978 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:23:46.978 INFO:teuthology.orchestra.run.smithi135.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:23:46.978 INFO:teuthology.orchestra.run.smithi135.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:23:46.978 INFO:teuthology.orchestra.run.smithi135.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:23:46.978 INFO:teuthology.orchestra.run.smithi135.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:23:46.978 INFO:teuthology.orchestra.run.smithi135.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:23:46.978 INFO:teuthology.orchestra.run.smithi135.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:23:46.979 INFO:teuthology.orchestra.run.smithi135.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:23:46.979 INFO:teuthology.orchestra.run.smithi135.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:23:47.492 INFO:teuthology.orchestra.run.smithi135.stdout:Discarding blocks...Done. 2024-03-29T11:23:47.494 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi135.front.sepia.ceph.com -o noatime 2024-03-29T11:23:47.494 DEBUG:teuthology.orchestra.run.smithi135:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:23:47.635 DEBUG:teuthology.orchestra.run.smithi135:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:23:47.671 DEBUG:teuthology.orchestra.run.smithi135:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:23:47.785 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:23:47.806+0000 7f0da560e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:23:47.788 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:23:47.808+0000 7f0da560e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:23:47.788 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:23:47.808+0000 7f0da560e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:23:47.788 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:23:47.808+0000 7f0da560e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:23:47.788 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:23:47.808+0000 7f0da560e640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:23:47.794 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:23:47.815+0000 7f0da560e640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:23:50.659 DEBUG:teuthology.orchestra.run.smithi135:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:23:50.690 DEBUG:teuthology.orchestra.run.smithi135:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:23:50.781 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:23:50.801+0000 7fba482b3640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:23:50.781 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:23:50.802+0000 7fba482b3640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:23:50.782 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:23:50.802+0000 7fba482b3640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:23:50.782 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:23:50.802+0000 7fba482b3640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:23:50.782 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:23:50.802+0000 7fba482b3640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:23:50.782 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:23:50.802+0000 7fba482b3640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:23:53.629 DEBUG:teuthology.orchestra.run.smithi135:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:23:53.661 DEBUG:teuthology.orchestra.run.smithi135:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:23:53.752 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:23:53.772+0000 7fd832022640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:23:53.752 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:23:53.772+0000 7fd832022640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:23:53.752 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:23:53.772+0000 7fd832022640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:23:53.753 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:23:53.772+0000 7fd832022640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:23:53.753 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:23:53.772+0000 7fd832022640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:23:53.753 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:23:53.773+0000 7fd832022640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:23:56.614 DEBUG:teuthology.orchestra.run.smithi135:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:23:56.642 DEBUG:teuthology.orchestra.run.smithi135:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:23:56.731 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:23:56.750+0000 7fb935a8b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:23:56.731 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:23:56.751+0000 7fb935a8b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:23:56.731 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:23:56.751+0000 7fb935a8b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:23:56.731 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:23:56.751+0000 7fb935a8b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:23:56.731 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:23:56.751+0000 7fb935a8b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:23:56.731 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:23:56.751+0000 7fb935a8b640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:23:59.592 DEBUG:teuthology.orchestra.run.smithi135:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:23:59.623 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:23:59.655 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:23:59.655 INFO:tasks.ceph:role: osd.4 2024-03-29T11:23:59.655 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi194.front.sepia.ceph.com 2024-03-29T11:23:59.655 DEBUG:teuthology.orchestra.run.smithi194:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:23:59.733 INFO:teuthology.orchestra.run.smithi194.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:23:59.733 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:23:59.733 INFO:teuthology.orchestra.run.smithi194.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:23:59.733 INFO:teuthology.orchestra.run.smithi194.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:23:59.733 INFO:teuthology.orchestra.run.smithi194.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:23:59.733 INFO:teuthology.orchestra.run.smithi194.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:23:59.733 INFO:teuthology.orchestra.run.smithi194.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:23:59.733 INFO:teuthology.orchestra.run.smithi194.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:23:59.733 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:23:59.734 INFO:teuthology.orchestra.run.smithi194.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:24:00.243 INFO:teuthology.orchestra.run.smithi194.stdout:Discarding blocks...Done. 2024-03-29T11:24:00.245 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi194.front.sepia.ceph.com -o noatime 2024-03-29T11:24:00.246 DEBUG:teuthology.orchestra.run.smithi194:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:24:00.389 DEBUG:teuthology.orchestra.run.smithi194:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:24:00.426 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:24:00.495 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:24:00.495 INFO:tasks.ceph:role: osd.5 2024-03-29T11:24:00.495 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi194.front.sepia.ceph.com 2024-03-29T11:24:00.496 DEBUG:teuthology.orchestra.run.smithi194:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:24:00.571 INFO:teuthology.orchestra.run.smithi194.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:24:00.571 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:24:00.571 INFO:teuthology.orchestra.run.smithi194.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:24:00.571 INFO:teuthology.orchestra.run.smithi194.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:24:00.572 INFO:teuthology.orchestra.run.smithi194.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:24:00.572 INFO:teuthology.orchestra.run.smithi194.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:24:00.572 INFO:teuthology.orchestra.run.smithi194.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:24:00.572 INFO:teuthology.orchestra.run.smithi194.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:24:00.572 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:24:00.572 INFO:teuthology.orchestra.run.smithi194.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:24:01.038 INFO:teuthology.orchestra.run.smithi194.stdout:Discarding blocks...Done. 2024-03-29T11:24:01.040 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi194.front.sepia.ceph.com -o noatime 2024-03-29T11:24:01.040 DEBUG:teuthology.orchestra.run.smithi194:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:24:01.128 DEBUG:teuthology.orchestra.run.smithi194:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:24:01.159 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:24:01.226 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:24:01.226 INFO:tasks.ceph:role: osd.6 2024-03-29T11:24:01.227 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi194.front.sepia.ceph.com 2024-03-29T11:24:01.227 DEBUG:teuthology.orchestra.run.smithi194:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:24:01.301 INFO:teuthology.orchestra.run.smithi194.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:24:01.301 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:24:01.301 INFO:teuthology.orchestra.run.smithi194.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:24:01.301 INFO:teuthology.orchestra.run.smithi194.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:24:01.301 INFO:teuthology.orchestra.run.smithi194.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:24:01.301 INFO:teuthology.orchestra.run.smithi194.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:24:01.301 INFO:teuthology.orchestra.run.smithi194.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:24:01.301 INFO:teuthology.orchestra.run.smithi194.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:24:01.301 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:24:01.301 INFO:teuthology.orchestra.run.smithi194.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:24:01.790 INFO:teuthology.orchestra.run.smithi194.stdout:Discarding blocks...Done. 2024-03-29T11:24:01.792 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi194.front.sepia.ceph.com -o noatime 2024-03-29T11:24:01.792 DEBUG:teuthology.orchestra.run.smithi194:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:24:01.922 DEBUG:teuthology.orchestra.run.smithi194:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:24:01.953 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:24:02.020 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:24:02.020 INFO:tasks.ceph:role: osd.7 2024-03-29T11:24:02.020 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi194.front.sepia.ceph.com 2024-03-29T11:24:02.020 DEBUG:teuthology.orchestra.run.smithi194:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:24:02.094 INFO:teuthology.orchestra.run.smithi194.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:24:02.094 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:24:02.095 INFO:teuthology.orchestra.run.smithi194.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:24:02.095 INFO:teuthology.orchestra.run.smithi194.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:24:02.095 INFO:teuthology.orchestra.run.smithi194.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:24:02.095 INFO:teuthology.orchestra.run.smithi194.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:24:02.095 INFO:teuthology.orchestra.run.smithi194.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:24:02.095 INFO:teuthology.orchestra.run.smithi194.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:24:02.095 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:24:02.095 INFO:teuthology.orchestra.run.smithi194.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:24:02.605 INFO:teuthology.orchestra.run.smithi194.stdout:Discarding blocks...Done. 2024-03-29T11:24:02.606 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi194.front.sepia.ceph.com -o noatime 2024-03-29T11:24:02.607 DEBUG:teuthology.orchestra.run.smithi194:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:24:02.740 DEBUG:teuthology.orchestra.run.smithi194:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:24:02.772 DEBUG:teuthology.orchestra.run.smithi194:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:24:02.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:02.892+0000 7f8f22213640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:24:02.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:02.892+0000 7f8f22213640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:24:02.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:02.893+0000 7f8f22213640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:24:02.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:02.893+0000 7f8f22213640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:24:02.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:02.893+0000 7f8f22213640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:24:02.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:02.893+0000 7f8f22213640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:24:05.719 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:24:05.751 DEBUG:teuthology.orchestra.run.smithi194:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:24:05.844 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:05.863+0000 7f14ce80f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:24:05.844 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:05.864+0000 7f14ce80f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:24:05.844 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:05.864+0000 7f14ce80f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:24:05.844 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:05.864+0000 7f14ce80f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:24:05.844 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:05.864+0000 7f14ce80f640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:24:05.844 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:05.864+0000 7f14ce80f640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:24:08.691 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:24:08.722 DEBUG:teuthology.orchestra.run.smithi194:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:24:08.813 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:08.833+0000 7fde066d9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:24:08.814 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:08.833+0000 7fde066d9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:24:08.814 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:08.833+0000 7fde066d9640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:24:08.814 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:08.833+0000 7fde066d9640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:24:08.814 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:08.833+0000 7fde066d9640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:24:08.814 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:08.834+0000 7fde066d9640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:24:11.666 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:24:11.697 DEBUG:teuthology.orchestra.run.smithi194:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:24:11.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:11.807+0000 7f638f868640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:24:11.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:11.807+0000 7f638f868640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:24:11.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:11.807+0000 7f638f868640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:24:11.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:11.807+0000 7f638f868640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:24:11.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:11.807+0000 7f638f868640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:24:11.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:11.807+0000 7f638f868640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:24:14.635 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:24:14.666 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:24:14.666 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-29T11:24:14.666 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:24:14.699 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-29T11:24:14.699 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:24:14.769 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-29T11:24:14.769 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:24:14.839 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-29T11:24:14.839 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:24:14.909 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-29T11:24:14.909 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:24:14.978 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-29T11:24:14.978 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:24:15.010 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-29T11:24:15.011 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:24:15.080 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-29T11:24:15.080 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:24:15.149 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-29T11:24:15.149 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:24:15.218 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-29T11:24:15.218 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:24:15.286 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-29T11:24:15.286 DEBUG:teuthology.orchestra.run.smithi135:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:24:15.303 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-29T11:24:15.304 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:24:15.342 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:24:15.342 DEBUG:teuthology.orchestra.run.smithi135:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:24:15.346 DEBUG:teuthology.orchestra.run.smithi194:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:24:15.385 INFO:teuthology.orchestra.run.smithi135.stdout:[mgr.y] 2024-03-29T11:24:15.386 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDApAZmPE3OGhAAto5UxL/gM3jq0AygTaTmWw== 2024-03-29T11:24:15.386 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.0] 2024-03-29T11:24:15.386 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDDpAZmyhYYMBAAINsEOO6DACGy1HMMyzoFZQ== 2024-03-29T11:24:15.386 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.1] 2024-03-29T11:24:15.386 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDGpAZmc/7TLxAAbLPXcJu40pfQ+g8J9TLi9Q== 2024-03-29T11:24:15.386 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.2] 2024-03-29T11:24:15.386 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDJpAZmwPcVLhAAdKFsp7t4dGKHFscCVAhlXw== 2024-03-29T11:24:15.386 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.3] 2024-03-29T11:24:15.386 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDMpAZmgJPILBAA5gbKsGxHZR2w3kGxXwFyxw== 2024-03-29T11:24:15.386 INFO:teuthology.orchestra.run.smithi135.stdout:[mgr.x] 2024-03-29T11:24:15.387 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDApAZmISb4HhAA8WOnyn+1GM6LmUqTlBUWOw== 2024-03-29T11:24:15.387 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.4] 2024-03-29T11:24:15.387 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDSpAZmwns6NRAANZH6fP9nvmLGbiRSmp9L8w== 2024-03-29T11:24:15.387 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.5] 2024-03-29T11:24:15.387 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDVpAZmcs2CMxAAFSXXEg4tGsLYDoDPMSObJg== 2024-03-29T11:24:15.387 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.6] 2024-03-29T11:24:15.387 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDYpAZmdpmvMRAAcPmeU9FeJnNeGTg12HPb6Q== 2024-03-29T11:24:15.387 INFO:teuthology.orchestra.run.smithi135.stdout:[osd.7] 2024-03-29T11:24:15.387 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDbpAZmSowhMBAAK2lymv583XR1waE+p05cpQ== 2024-03-29T11:24:15.387 INFO:teuthology.orchestra.run.smithi135.stdout:[client.0] 2024-03-29T11:24:15.387 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDApAZm9UukIRAA0ZGdsxP4GAXxTxo7Yrl+cg== 2024-03-29T11:24:15.387 INFO:teuthology.orchestra.run.smithi135.stdout:[client.1] 2024-03-29T11:24:15.387 INFO:teuthology.orchestra.run.smithi135.stdout: key = AQDApAZmQc8RJRAAORq1MUQCPRX/IORMJ4OSfQ== 2024-03-29T11:24:15.408 INFO:teuthology.orchestra.run.smithi194.stdout:[mgr.y] 2024-03-29T11:24:15.408 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDApAZmPE3OGhAAto5UxL/gM3jq0AygTaTmWw== 2024-03-29T11:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.0] 2024-03-29T11:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDDpAZmyhYYMBAAINsEOO6DACGy1HMMyzoFZQ== 2024-03-29T11:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.1] 2024-03-29T11:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDGpAZmc/7TLxAAbLPXcJu40pfQ+g8J9TLi9Q== 2024-03-29T11:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.2] 2024-03-29T11:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDJpAZmwPcVLhAAdKFsp7t4dGKHFscCVAhlXw== 2024-03-29T11:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.3] 2024-03-29T11:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDMpAZmgJPILBAA5gbKsGxHZR2w3kGxXwFyxw== 2024-03-29T11:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout:[mgr.x] 2024-03-29T11:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDApAZmISb4HhAA8WOnyn+1GM6LmUqTlBUWOw== 2024-03-29T11:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.4] 2024-03-29T11:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDSpAZmwns6NRAANZH6fP9nvmLGbiRSmp9L8w== 2024-03-29T11:24:15.409 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.5] 2024-03-29T11:24:15.410 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDVpAZmcs2CMxAAFSXXEg4tGsLYDoDPMSObJg== 2024-03-29T11:24:15.410 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.6] 2024-03-29T11:24:15.410 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDYpAZmdpmvMRAAcPmeU9FeJnNeGTg12HPb6Q== 2024-03-29T11:24:15.410 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.7] 2024-03-29T11:24:15.410 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDbpAZmSowhMBAAK2lymv583XR1waE+p05cpQ== 2024-03-29T11:24:15.410 INFO:teuthology.orchestra.run.smithi194.stdout:[client.0] 2024-03-29T11:24:15.410 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDApAZm9UukIRAA0ZGdsxP4GAXxTxo7Yrl+cg== 2024-03-29T11:24:15.410 INFO:teuthology.orchestra.run.smithi194.stdout:[client.1] 2024-03-29T11:24:15.410 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQDApAZmQc8RJRAAORq1MUQCPRX/IORMJ4OSfQ== 2024-03-29T11:24:15.411 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:24:15.431 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:24:15.503 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:24:15.506 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:24:15.558 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:24:15.560 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:24:15.611 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:24:15.614 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:24:15.665 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:24:15.668 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:24:15.719 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:24:15.721 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:24:15.772 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:24:15.775 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:24:15.828 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:24:15.830 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:24:15.882 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:24:15.884 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:24:15.935 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:24:15.938 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:24:15.989 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:24:15.992 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:24:16.047 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:24:16.049 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:24:16.103 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:24:16.103 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:24:16.130 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:24:16.227 DEBUG:teuthology.orchestra.run.smithi135:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:24:16.253 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:24:16.320 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:24:16.418 DEBUG:teuthology.orchestra.run.smithi135:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:24:16.445 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:24:16.472 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:24:16.570 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:24:16.597 DEBUG:teuthology.orchestra.run.smithi135:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:24:16.599 DEBUG:teuthology.orchestra.run.smithi194:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:24:16.652 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:24:16.652 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:24:16.652 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:24:16.654 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:24:16.654 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:24:16.655 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:24:16.656 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:24:16.657 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:24:16.657 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:24:16.694 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:24:16.695 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:24:16.695 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:24:16.695 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:24:16.697 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:24:16.697 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:24:16.698 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:24:16.700 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:24:16.700 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:24:16.700 DEBUG:teuthology.orchestra.run.smithi135:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:24:16.801 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:16.819+0000 7f7771339640 10 monclient: get_monmap_and_config 2024-03-29T11:24:16.801 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:16.819+0000 7f7771339640 10 monclient: build_initial_monmap 2024-03-29T11:24:16.801 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:16.819+0000 7f7771339640 10 monclient: monmap: 2024-03-29T11:24:16.801 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2024-03-29T11:24:16.801 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:16.802 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-03-29T11:24:16.821328+0000 2024-03-29T11:24:16.802 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-03-29T11:24:16.821328+0000 2024-03-29T11:24:16.802 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:16.802 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:16.802 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:16.802 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:24:16.802 INFO:teuthology.orchestra.run.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:24:16.802 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-03-29T11:24:16.803 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:16.820+0000 7f7771339640 1 Processor -- start 2024-03-29T11:24:16.803 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:16.820+0000 7f7771339640 1 -- start start 2024-03-29T11:24:16.803 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:16.820+0000 7f7771339640 10 monclient: init 2024-03-29T11:24:16.803 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:16.820+0000 7f7771339640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:16.804 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:16.820+0000 7f7771339640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:16.804 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:16.820+0000 7f7771339640 10 monclient(hunting): picked mon.noname-b con 0x7f776c09ac40 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:16.804 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:16.820+0000 7f7771339640 10 monclient(hunting): picked mon.noname-c con 0x7f776c09c870 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:16.804 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:16.820+0000 7f7771339640 10 monclient(hunting): picked mon.noname-a con 0x7f776c0b4360 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:16.804 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:16.820+0000 7f7771339640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776c09dec0 con 0x7f776c0b4360 2024-03-29T11:24:16.804 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:16.820+0000 7f7771339640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776c09da30 con 0x7f776c09c870 2024-03-29T11:24:16.804 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:16.820+0000 7f7771339640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776c0b39c0 con 0x7f776c09ac40 2024-03-29T11:24:16.804 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:16.820+0000 7f7771339640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:16.804 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:16.820+0000 7f7771339640 10 monclient(hunting): authenticate will time out at 764.189991s 2024-03-29T11:24:16.804 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:16.821+0000 7f776affd640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f776c09ac40 0x7f776c0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:35048/0 (socket says 172.21.15.135:35048) 2024-03-29T11:24:16.804 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:16.821+0000 7f776affd640 1 -- 172.21.15.135:0/3943135389 learned_addr learned my addr 172.21.15.135:0/3943135389 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:24:19.803 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:19.821+0000 7f77697fa640 10 monclient(hunting): tick 2024-03-29T11:24:19.803 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:19.821+0000 7f77697fa640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:24:19.804 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:19.821+0000 7f77697fa640 1 monclient(hunting): continuing hunt 2024-03-29T11:24:19.804 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:19.821+0000 7f77697fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:24:19.804 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:19.821+0000 7f77697fa640 1 -- 172.21.15.135:0/3943135389 >> v1:172.21.15.135:6790/0 conn(0x7f776c09ac40 legacy=0x7f776c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:19.804 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:19.821+0000 7f77697fa640 1 -- 172.21.15.135:0/3943135389 >> v1:172.21.15.194:6789/0 conn(0x7f776c09c870 legacy=0x7f776c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:19.804 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:19.821+0000 7f77697fa640 1 -- 172.21.15.135:0/3943135389 >> v1:172.21.15.135:6789/0 conn(0x7f776c0b4360 legacy=0x7f776c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:19.804 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:19.821+0000 7f77697fa640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:19.804 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:19.821+0000 7f77697fa640 10 monclient(hunting): picked mon.noname-c con 0x7f774c0036e0 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:19.804 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:19.821+0000 7f77697fa640 10 monclient(hunting): picked mon.noname-b con 0x7f774c0072d0 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:19.804 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:19.821+0000 7f77697fa640 10 monclient(hunting): picked mon.noname-a con 0x7f774c00afd0 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:19.804 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:19.821+0000 7f77697fa640 1 -- 172.21.15.135:0/3943135389 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776c0b5160 con 0x7f774c00afd0 2024-03-29T11:24:19.804 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:19.821+0000 7f77697fa640 1 -- 172.21.15.135:0/3943135389 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f774c00fc80 con 0x7f774c0036e0 2024-03-29T11:24:19.804 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:19.821+0000 7f77697fa640 1 -- 172.21.15.135:0/3943135389 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f774c010ee0 con 0x7f774c0072d0 2024-03-29T11:24:19.804 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:19.821+0000 7f77697fa640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:21.976 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.991+0000 7f776a7fc640 1 -- 172.21.15.135:0/3943135389 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4255994330 0 0) 0x7f776c0b5160 con 0x7f774c00afd0 2024-03-29T11:24:21.976 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f776a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:21.976 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f776a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:21.976 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f776a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:21.976 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f776a7fc640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:24:21.976 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f776a7fc640 1 -- 172.21.15.135:0/3943135389 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7758003520 con 0x7f774c00afd0 2024-03-29T11:24:21.976 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f776a7fc640 1 -- 172.21.15.135:0/3943135389 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2521481648 0 0) 0x7f774c00fc80 con 0x7f774c0036e0 2024-03-29T11:24:21.976 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f776a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:21.976 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f776a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:21.977 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f776a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:21.977 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f776a7fc640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:24:21.977 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f776a7fc640 1 -- 172.21.15.135:0/3943135389 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f776c0b5160 con 0x7f774c0036e0 2024-03-29T11:24:21.977 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f776a7fc640 1 -- 172.21.15.135:0/3943135389 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2401244572 0 0) 0x7f776c0b5160 con 0x7f774c0036e0 2024-03-29T11:24:21.977 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f776a7fc640 1 -- 172.21.15.135:0/3943135389 >> v1:172.21.15.135:6790/0 conn(0x7f774c0072d0 legacy=0x7f774c009760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:21.977 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f776a7fc640 1 -- 172.21.15.135:0/3943135389 >> v1:172.21.15.135:6789/0 conn(0x7f774c00afd0 legacy=0x7f774c00d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:21.977 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f776a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:21.977 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f776a7fc640 1 monclient: found mon.noname-c 2024-03-29T11:24:21.977 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f776a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:21.977 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f776a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:24:21.978 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f776a7fc640 1 -- 172.21.15.135:0/3943135389 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f774c012140 con 0x7f774c0036e0 2024-03-29T11:24:21.978 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f776a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:21.978 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f776a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:21.978 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f776a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:21.978 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f776a7fc640 1 -- 172.21.15.135:0/3943135389 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f775c001610 con 0x7f774c0036e0 2024-03-29T11:24:21.978 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f776a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:21.978 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f776a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:21.978 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f776a7fc640 10 monclient: dump: 2024-03-29T11:24:21.978 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-03-29T11:24:21.978 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:21.978 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:21.978 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:21.978 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:21.979 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:21.979 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:21.979 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:21.979 INFO:teuthology.orchestra.run.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:21.979 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-03-29T11:24:21.979 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f7771339640 5 monclient: authenticate success, global_id 4105 2024-03-29T11:24:21.979 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.992+0000 7f7771339640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:21.979 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.993+0000 7f776a7fc640 1 -- 172.21.15.135:0/3943135389 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f775c001770 con 0x7f774c0036e0 2024-03-29T11:24:21.979 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.993+0000 7f776a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:21.979 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.993+0000 7f776a7fc640 1 -- 172.21.15.135:0/3943135389 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f775c001b80 con 0x7f774c0036e0 2024-03-29T11:24:21.979 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.993+0000 7f776a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:21.979 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.993+0000 7f776a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:21.979 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.993+0000 7f776a7fc640 10 monclient: dump: 2024-03-29T11:24:21.979 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-03-29T11:24:21.979 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:21.979 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:21.980 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:21.980 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:21.980 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:21.980 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:21.980 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:21.980 INFO:teuthology.orchestra.run.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:21.980 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-03-29T11:24:21.980 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.993+0000 7f7771339640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:21.980 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.993+0000 7f7771339640 4 set_mon_vals no callback set 2024-03-29T11:24:21.980 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.994+0000 7f7771339640 10 monclient: shutdown 2024-03-29T11:24:21.980 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.994+0000 7f7771339640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:21.980 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.994+0000 7f7771339640 1 -- 172.21.15.135:0/3943135389 >> v1:172.21.15.194:6789/0 conn(0x7f774c0036e0 legacy=0x7f774c005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:21.980 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.994+0000 7f7771339640 1 -- 172.21.15.135:0/3943135389 shutdown_connections 2024-03-29T11:24:21.981 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.994+0000 7f7771339640 1 -- 172.21.15.135:0/3943135389 >> 172.21.15.135:0/3943135389 conn(0x7f776c066b00 msgr2=0x7f776c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:21.981 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.994+0000 7f7771339640 1 -- 172.21.15.135:0/3943135389 shutdown_connections 2024-03-29T11:24:21.981 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.994+0000 7f7771339640 1 -- 172.21.15.135:0/3943135389 wait complete. 2024-03-29T11:24:21.989 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.995+0000 7f7771339640 10 monclient: build_initial_monmap 2024-03-29T11:24:21.989 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.995+0000 7f7771339640 10 monclient: monmap: 2024-03-29T11:24:21.989 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2024-03-29T11:24:21.989 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:21.989 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 0.000000 2024-03-29T11:24:21.989 INFO:teuthology.orchestra.run.smithi135.stderr:created 0.000000 2024-03-29T11:24:21.989 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:21.990 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:21.990 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:21.990 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:24:21.990 INFO:teuthology.orchestra.run.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:24:21.990 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-03-29T11:24:21.990 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.995+0000 7f7771339640 1 librados: starting msgr at 2024-03-29T11:24:21.990 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.995+0000 7f7771339640 1 librados: starting objecter 2024-03-29T11:24:21.990 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.995+0000 7f7771339640 1 Processor -- start 2024-03-29T11:24:21.990 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.996+0000 7f7771339640 1 -- start start 2024-03-29T11:24:21.990 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.996+0000 7f7771339640 1 librados: setting wanted keys 2024-03-29T11:24:21.990 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.996+0000 7f7771339640 1 librados: calling monclient init 2024-03-29T11:24:21.990 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.996+0000 7f7771339640 10 monclient: init 2024-03-29T11:24:21.990 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.997+0000 7f7771339640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:21.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.997+0000 7f7771339640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:21.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.997+0000 7f7771339640 10 monclient(hunting): picked mon.noname-c con 0x7f774c0036e0 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:21.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.997+0000 7f7771339640 10 monclient(hunting): picked mon.noname-b con 0x7f774c0072d0 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:21.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.997+0000 7f7771339640 10 monclient(hunting): picked mon.noname-a con 0x7f774c00afd0 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:21.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.997+0000 7f7771339640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776c142530 con 0x7f774c00afd0 2024-03-29T11:24:21.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.997+0000 7f7771339640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776c1df750 con 0x7f774c0072d0 2024-03-29T11:24:21.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.997+0000 7f7771339640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776c142780 con 0x7f774c0036e0 2024-03-29T11:24:21.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.997+0000 7f7771339640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:21.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.997+0000 7f776affd640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f774c0036e0 0x7f776c1de930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:35096/0 (socket says 172.21.15.135:35096) 2024-03-29T11:24:21.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.997+0000 7f776affd640 1 -- 172.21.15.135:0/3238253855 learned_addr learned my addr 172.21.15.135:0/3238253855 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:24:21.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.997+0000 7f7771339640 10 monclient(hunting): authenticate will time out at 769.366286s 2024-03-29T11:24:21.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.997+0000 7f7763fff640 10 client.?.objecter ms_handle_connect 0x7f774c0072d0 2024-03-29T11:24:21.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.997+0000 7f7763fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:21.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.997+0000 7f7763fff640 10 client.?.objecter ms_handle_connect 0x7f774c0036e0 2024-03-29T11:24:21.991 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.997+0000 7f7763fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:21.992 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.997+0000 7f7763fff640 10 client.?.objecter ms_handle_connect 0x7f774c00afd0 2024-03-29T11:24:21.992 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.997+0000 7f7763fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:21.992 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.997+0000 7f7763fff640 1 -- 172.21.15.135:0/3238253855 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778186575 0 0) 0x7f776c142530 con 0x7f774c00afd0 2024-03-29T11:24:21.992 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.997+0000 7f7763fff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:21.992 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.997+0000 7f7763fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:21.992 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.997+0000 7f7763fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:21.992 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.997+0000 7f7763fff640 10 monclient(hunting): my global_id is 4110 2024-03-29T11:24:21.992 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.997+0000 7f7763fff640 1 -- 172.21.15.135:0/3238253855 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f774c011830 con 0x7f774c00afd0 2024-03-29T11:24:21.992 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7763fff640 1 -- 172.21.15.135:0/3238253855 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2283425488 0 0) 0x7f774c011830 con 0x7f774c00afd0 2024-03-29T11:24:21.992 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7763fff640 1 -- 172.21.15.135:0/3238253855 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f776c142530 con 0x7f774c00afd0 2024-03-29T11:24:21.992 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7763fff640 1 -- 172.21.15.135:0/3238253855 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f7764001ad0 con 0x7f774c00afd0 2024-03-29T11:24:21.992 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7763fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:21.993 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7763fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:21.993 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7763fff640 10 monclient(hunting): dump: 2024-03-29T11:24:21.993 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-03-29T11:24:21.993 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:21.993 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:21.993 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:21.993 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:21.993 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:21.993 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:21.993 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:21.993 INFO:teuthology.orchestra.run.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:21.993 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-03-29T11:24:21.993 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7763fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:21.993 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7763fff640 1 -- 172.21.15.135:0/3238253855 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2548661526 0 0) 0x7f776c142530 con 0x7f774c00afd0 2024-03-29T11:24:21.993 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7763fff640 1 -- 172.21.15.135:0/3238253855 >> v1:172.21.15.135:6790/0 conn(0x7f774c0036e0 legacy=0x7f776c1de930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:21.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7763fff640 1 -- 172.21.15.135:0/3238253855 >> v1:172.21.15.194:6789/0 conn(0x7f774c0072d0 legacy=0x7f776c0b6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:21.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7763fff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:21.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7763fff640 1 monclient: found mon.a 2024-03-29T11:24:21.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7763fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:21.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7763fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:21.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7763fff640 1 -- 172.21.15.135:0/3238253855 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f776c142a60 con 0x7f774c00afd0 2024-03-29T11:24:21.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7763fff640 10 monclient: _finish_auth 0 2024-03-29T11:24:21.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7763fff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:21.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7763fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:21.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7771339640 5 monclient: authenticate success, global_id 4110 2024-03-29T11:24:21.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7771339640 10 monclient: _renew_subs 2024-03-29T11:24:21.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7771339640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:21.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7771339640 1 -- 172.21.15.135:0/3238253855 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f776c1415a0 con 0x7f774c00afd0 2024-03-29T11:24:21.994 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7771339640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:21.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7771339640 10 monclient: _renew_subs 2024-03-29T11:24:21.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7771339640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:21.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7771339640 1 -- 172.21.15.135:0/3238253855 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f776c143060 con 0x7f774c00afd0 2024-03-29T11:24:21.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:21.998+0000 7f7771339640 1 librados: init done 2024-03-29T11:24:21.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.000+0000 7f7771339640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:21.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.000+0000 7f7771339640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:21.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.000+0000 7f7771339640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:21.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.000+0000 7f7771339640 1 -- 172.21.15.135:0/3238253855 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f776c09b020 con 0x7f774c00afd0 2024-03-29T11:24:21.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.000+0000 7f7763fff640 1 -- 172.21.15.135:0/3238253855 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7764001fe0 con 0x7f774c00afd0 2024-03-29T11:24:21.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.000+0000 7f7763fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:21.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.000+0000 7f7769ffb640 4 set_mon_vals no callback set 2024-03-29T11:24:21.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.000+0000 7f7763fff640 1 -- 172.21.15.135:0/3238253855 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f7764003b10 con 0x7f774c00afd0 2024-03-29T11:24:21.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.000+0000 7f7763fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:21.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.000+0000 7f7763fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:21.995 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.000+0000 7f7763fff640 10 monclient: dump: 2024-03-29T11:24:21.996 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-03-29T11:24:21.996 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:21.996 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:21.996 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:21.996 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:21.996 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:21.996 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:21.996 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:21.996 INFO:teuthology.orchestra.run.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:21.996 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-03-29T11:24:21.996 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.000+0000 7f7763fff640 1 -- 172.21.15.135:0/3238253855 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7764004000 con 0x7f774c00afd0 2024-03-29T11:24:21.996 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.000+0000 7f7763fff640 1 -- 172.21.15.135:0/3238253855 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 96007774 0 0) 0x7f77640055d0 con 0x7f774c00afd0 2024-03-29T11:24:21.996 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.000+0000 7f7763fff640 10 client.4110.objecter ms_dispatch 0x7f776c000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:24:21.996 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.000+0000 7f7763fff640 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:24:21.997 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.000+0000 7f7763fff640 3 client.4110.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:24:21.997 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.000+0000 7f7763fff640 20 client.4110.objecter dump_active .. 0 homeless 2024-03-29T11:24:21.997 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.003+0000 7f7763fff640 1 -- 172.21.15.135:0/3238253855 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f77640059d0 con 0x7f774c00afd0 2024-03-29T11:24:21.997 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.003+0000 7f7763fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:21.997 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.003+0000 7f7763fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:22.055 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.073+0000 7f7771339640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:24:22.056 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.073+0000 7f7771339640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:24:22.056 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.073+0000 7f7771339640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:22.056 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.073+0000 7f7771339640 1 -- 172.21.15.135:0/3238253855 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f776c04f730 con 0x7f774c00afd0 2024-03-29T11:24:22.057 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.075+0000 7f7763fff640 1 -- 172.21.15.135:0/3238253855 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f7764004390 con 0x7f774c00afd0 2024-03-29T11:24:22.057 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.075+0000 7f7763fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:24:22.057 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.075+0000 7f7763fff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:24:22.058 INFO:teuthology.orchestra.run.smithi135.stderr:adjusted tunables profile to default 2024-03-29T11:24:22.059 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.076+0000 7f7771339640 10 librados: watch_flush enter 2024-03-29T11:24:22.059 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.076+0000 7f7771339640 10 librados: watch_flush exit 2024-03-29T11:24:22.059 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.076+0000 7f7771339640 20 client.4110.objecter shutdown clearing up homeless session... 2024-03-29T11:24:22.060 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.076+0000 7f7771339640 10 client.4110.objecter successfully canceled tick 2024-03-29T11:24:22.060 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.077+0000 7f7771339640 10 monclient: shutdown 2024-03-29T11:24:22.060 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.077+0000 7f7771339640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:22.060 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.077+0000 7f7771339640 1 -- 172.21.15.135:0/3238253855 >> v1:172.21.15.135:6789/0 conn(0x7f774c00afd0 legacy=0x7f776c1df040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:22.060 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.077+0000 7f7771339640 1 -- 172.21.15.135:0/3238253855 shutdown_connections 2024-03-29T11:24:22.060 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.077+0000 7f7771339640 1 -- 172.21.15.135:0/3238253855 >> 172.21.15.135:0/3238253855 conn(0x7f776c066b00 msgr2=0x7f776c1cbd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:22.060 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.077+0000 7f7771339640 1 -- 172.21.15.135:0/3238253855 shutdown_connections 2024-03-29T11:24:22.061 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.077+0000 7f7771339640 1 -- 172.21.15.135:0/3238253855 wait complete. 2024-03-29T11:24:22.061 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:22.077+0000 7f7771339640 1 librados: shutdown 2024-03-29T11:24:22.071 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:24:22.071 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:24:22.071 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-29T11:24:22.071 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:24:22.099 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-29T11:24:22.099 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:24:22.167 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-29T11:24:22.167 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:24:22.234 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2024-03-29T11:24:22.235 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:24:22.302 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-29T11:24:22.302 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:24:22.330 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-29T11:24:22.330 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:24:22.396 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-29T11:24:22.397 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:24:22.457 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:22.475+0000 7f992383e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:24:22.463 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-29T11:24:22.463 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:24:22.494 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-29T11:24:22.512+0000 7fe5b562c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:24:22.510 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:22.529+0000 7f992383e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:24:22.520 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new d8d8500d-c94b-47b0-82f0-8e2c681dd656 0 2024-03-29T11:24:22.550 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-29T11:24:22.569+0000 7fe5b562c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:24:22.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.668+0000 7ff60a16b640 10 monclient: get_monmap_and_config 2024-03-29T11:24:22.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.668+0000 7ff60a16b640 10 monclient: build_initial_monmap 2024-03-29T11:24:22.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.668+0000 7ff60a16b640 10 monclient: monmap: 2024-03-29T11:24:22.650 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-29T11:24:22.650 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:22.650 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:24:22.669538+0000 2024-03-29T11:24:22.650 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:24:22.669538+0000 2024-03-29T11:24:22.650 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:22.650 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:22.650 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:22.650 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:24:22.651 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:24:22.651 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:22.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.668+0000 7ff60a16b640 1 Processor -- start 2024-03-29T11:24:22.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.668+0000 7ff60a16b640 1 -- start start 2024-03-29T11:24:22.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.668+0000 7ff60a16b640 10 monclient: init 2024-03-29T11:24:22.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.669+0000 7ff60a16b640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:22.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.669+0000 7ff60a16b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:22.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.669+0000 7ff60a16b640 10 monclient(hunting): picked mon.noname-a con 0x7ff60409ac40 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:22.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.669+0000 7ff60a16b640 10 monclient(hunting): picked mon.noname-b con 0x7ff60409c870 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:22.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.669+0000 7ff60a16b640 10 monclient(hunting): picked mon.noname-c con 0x7ff6040b4360 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:22.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.669+0000 7ff60a16b640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff60409dec0 con 0x7ff60409ac40 2024-03-29T11:24:22.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.669+0000 7ff60a16b640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff60409da30 con 0x7ff6040b4360 2024-03-29T11:24:22.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.669+0000 7ff60a16b640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6040b39c0 con 0x7ff60409c870 2024-03-29T11:24:22.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.669+0000 7ff60a16b640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:22.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.669+0000 7ff60a16b640 10 monclient(hunting): authenticate will time out at 769.955673s 2024-03-29T11:24:22.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.669+0000 7ff603fff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7ff6040b4360 0x7ff60414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:57272/0 (socket says 172.21.15.194:57272) 2024-03-29T11:24:22.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.669+0000 7ff603fff640 1 -- 172.21.15.194:0/4212545790 learned_addr learned my addr 172.21.15.194:0/4212545790 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-29T11:24:22.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.669+0000 7ff6027fc640 1 -- 172.21.15.194:0/4212545790 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3714585639 0 0) 0x7ff60409da30 con 0x7ff6040b4360 2024-03-29T11:24:22.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:22.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:22.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:22.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 10 monclient(hunting): my global_id is 4111 2024-03-29T11:24:22.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 1 -- 172.21.15.194:0/4212545790 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5ec003520 con 0x7ff6040b4360 2024-03-29T11:24:22.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 1 -- 172.21.15.194:0/4212545790 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4105918235 0 0) 0x7ff6040b39c0 con 0x7ff60409c870 2024-03-29T11:24:22.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:22.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:22.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:22.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:24:22.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 1 -- 172.21.15.194:0/4212545790 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff60409da30 con 0x7ff60409c870 2024-03-29T11:24:22.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 1 -- 172.21.15.194:0/4212545790 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045678710 0 0) 0x7ff5f8001110 con 0x7ff60409ac40 2024-03-29T11:24:22.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:22.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:22.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:22.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:24:22.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 1 -- 172.21.15.194:0/4212545790 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6040b39c0 con 0x7ff60409ac40 2024-03-29T11:24:22.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 1 -- 172.21.15.194:0/4212545790 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3168043773 0 0) 0x7ff5ec003520 con 0x7ff6040b4360 2024-03-29T11:24:22.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 1 -- 172.21.15.194:0/4212545790 >> v1:172.21.15.135:6790/0 conn(0x7ff60409c870 legacy=0x7ff60413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:22.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 1 -- 172.21.15.194:0/4212545790 >> v1:172.21.15.135:6789/0 conn(0x7ff60409ac40 legacy=0x7ff6040b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:22.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:22.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 1 monclient: found mon.noname-c 2024-03-29T11:24:22.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:22.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:24:22.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 1 -- 172.21.15.194:0/4212545790 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6040b5160 con 0x7ff6040b4360 2024-03-29T11:24:22.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:22.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:22.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:22.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 1 -- 172.21.15.194:0/4212545790 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7ff5fc001a10 con 0x7ff6040b4360 2024-03-29T11:24:22.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff60a16b640 5 monclient: authenticate success, global_id 4111 2024-03-29T11:24:22.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff60a16b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:22.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:22.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:22.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 10 monclient: dump: 2024-03-29T11:24:22.662 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:22.662 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:22.662 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:22.662 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:22.662 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:22.662 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:22.663 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:22.663 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:22.663 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:22.663 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:22.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff60a16b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:22.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 1 -- 172.21.15.194:0/4212545790 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5fc001bf0 con 0x7ff6040b4360 2024-03-29T11:24:22.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:22.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 1 -- 172.21.15.194:0/4212545790 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7ff5fc002000 con 0x7ff6040b4360 2024-03-29T11:24:22.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff60a16b640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:22.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff60a16b640 4 set_mon_vals no callback set 2024-03-29T11:24:22.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:22.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:22.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff6027fc640 10 monclient: dump: 2024-03-29T11:24:22.663 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:22.663 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:22.664 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:22.664 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:22.664 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:22.664 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:22.664 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:22.664 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:22.664 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:22.664 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:22.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff60a16b640 10 monclient: shutdown 2024-03-29T11:24:22.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff60a16b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:22.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff60a16b640 1 -- 172.21.15.194:0/4212545790 >> v1:172.21.15.194:6789/0 conn(0x7ff6040b4360 legacy=0x7ff60414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:22.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff60a16b640 1 -- 172.21.15.194:0/4212545790 shutdown_connections 2024-03-29T11:24:22.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.670+0000 7ff60a16b640 1 -- 172.21.15.194:0/4212545790 >> 172.21.15.194:0/4212545790 conn(0x7ff604066b00 msgr2=0x7ff604066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:22.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.671+0000 7ff60a16b640 1 -- 172.21.15.194:0/4212545790 shutdown_connections 2024-03-29T11:24:22.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.671+0000 7ff60a16b640 1 -- 172.21.15.194:0/4212545790 wait complete. 2024-03-29T11:24:22.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.671+0000 7ff60a16b640 10 monclient: build_initial_monmap 2024-03-29T11:24:22.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.671+0000 7ff60a16b640 10 monclient: monmap: 2024-03-29T11:24:22.665 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-29T11:24:22.665 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:22.665 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 0.000000 2024-03-29T11:24:22.665 INFO:teuthology.orchestra.run.smithi194.stderr:created 0.000000 2024-03-29T11:24:22.665 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:22.665 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:22.665 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:22.665 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:24:22.665 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:24:22.665 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:22.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.671+0000 7ff60a16b640 1 librados: starting msgr at 2024-03-29T11:24:22.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.671+0000 7ff60a16b640 1 librados: starting objecter 2024-03-29T11:24:22.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.671+0000 7ff60a16b640 1 Processor -- start 2024-03-29T11:24:22.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.672+0000 7ff60a16b640 1 -- start start 2024-03-29T11:24:22.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.672+0000 7ff60a16b640 1 librados: setting wanted keys 2024-03-29T11:24:22.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.672+0000 7ff60a16b640 1 librados: calling monclient init 2024-03-29T11:24:22.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.672+0000 7ff60a16b640 10 monclient: init 2024-03-29T11:24:22.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.672+0000 7ff60a16b640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:22.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.672+0000 7ff60a16b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:22.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.672+0000 7ff60a16b640 10 monclient(hunting): picked mon.noname-a con 0x7ff60409ac40 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:22.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.672+0000 7ff60a16b640 10 monclient(hunting): picked mon.noname-c con 0x7ff6041454c0 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:22.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.672+0000 7ff60a16b640 10 monclient(hunting): picked mon.noname-b con 0x7ff604144ec0 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:22.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.672+0000 7ff60a16b640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6041379b0 con 0x7ff60409ac40 2024-03-29T11:24:22.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.672+0000 7ff60a16b640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff604140a70 con 0x7ff604144ec0 2024-03-29T11:24:22.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.672+0000 7ff60a16b640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff604140d50 con 0x7ff6041454c0 2024-03-29T11:24:22.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.672+0000 7ff60a16b640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:22.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.672+0000 7ff60a16b640 10 monclient(hunting): authenticate will time out at 769.958819s 2024-03-29T11:24:22.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.672+0000 7ff603fff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7ff604144ec0 0x7ff6040b0620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:57280/0 (socket says 172.21.15.194:57280) 2024-03-29T11:24:22.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.672+0000 7ff603fff640 1 -- 172.21.15.194:0/3393455700 learned_addr learned my addr 172.21.15.194:0/3393455700 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-29T11:24:22.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.672+0000 7ff5ebfff640 10 client.?.objecter ms_handle_connect 0x7ff604144ec0 2024-03-29T11:24:22.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.672+0000 7ff5ebfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:22.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.672+0000 7ff5ebfff640 10 client.?.objecter ms_handle_connect 0x7ff60409ac40 2024-03-29T11:24:22.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.672+0000 7ff5ebfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:22.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.672+0000 7ff5ebfff640 10 client.?.objecter ms_handle_connect 0x7ff6041454c0 2024-03-29T11:24:22.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.672+0000 7ff5ebfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:22.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 1 -- 172.21.15.194:0/3393455700 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3330883241 0 0) 0x7ff604140a70 con 0x7ff604144ec0 2024-03-29T11:24:22.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:22.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:22.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:22.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:24:22.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 1 -- 172.21.15.194:0/3393455700 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5f0004eb0 con 0x7ff604144ec0 2024-03-29T11:24:22.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 1 -- 172.21.15.194:0/3393455700 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039031977 0 0) 0x7ff6041379b0 con 0x7ff60409ac40 2024-03-29T11:24:22.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:22.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:22.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:22.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:24:22.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 1 -- 172.21.15.194:0/3393455700 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff604140a70 con 0x7ff60409ac40 2024-03-29T11:24:22.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 1 -- 172.21.15.194:0/3393455700 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3039551220 0 0) 0x7ff604140d50 con 0x7ff6041454c0 2024-03-29T11:24:22.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:22.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:22.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:22.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient(hunting): my global_id is 4109 2024-03-29T11:24:22.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 1 -- 172.21.15.194:0/3393455700 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6041379b0 con 0x7ff6041454c0 2024-03-29T11:24:22.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 1 -- 172.21.15.194:0/3393455700 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3260276153 0 0) 0x7ff5f0004eb0 con 0x7ff604144ec0 2024-03-29T11:24:22.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 1 -- 172.21.15.194:0/3393455700 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff604140d50 con 0x7ff604144ec0 2024-03-29T11:24:22.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 1 -- 172.21.15.194:0/3393455700 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7ff5fc001f40 con 0x7ff604144ec0 2024-03-29T11:24:22.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:22.669 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:22.669 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient(hunting): dump: 2024-03-29T11:24:22.669 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:22.669 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:22.669 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:22.669 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:22.669 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:22.669 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:22.669 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:22.669 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:22.669 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:22.669 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:22.669 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:22.669 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 1 -- 172.21.15.194:0/3393455700 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1115416795 0 0) 0x7ff604140a70 con 0x7ff60409ac40 2024-03-29T11:24:22.669 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 1 -- 172.21.15.194:0/3393455700 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5f0004eb0 con 0x7ff60409ac40 2024-03-29T11:24:22.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 1 -- 172.21.15.194:0/3393455700 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4126922787 0 0) 0x7ff6041379b0 con 0x7ff6041454c0 2024-03-29T11:24:22.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 1 -- 172.21.15.194:0/3393455700 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff604140a70 con 0x7ff6041454c0 2024-03-29T11:24:22.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 1 -- 172.21.15.194:0/3393455700 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7ff5f8001860 con 0x7ff60409ac40 2024-03-29T11:24:22.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:22.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:22.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient(hunting): dump: 2024-03-29T11:24:22.670 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:22.670 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:22.670 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:22.670 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:22.670 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:22.670 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:22.670 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:22.670 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:22.671 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:22.671 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:22.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 1 -- 172.21.15.194:0/3393455700 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7ff5f4001710 con 0x7ff6041454c0 2024-03-29T11:24:22.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:22.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:22.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient(hunting): dump: 2024-03-29T11:24:22.671 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:22.671 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:22.671 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:22.671 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:22.671 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:22.671 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:22.671 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:22.671 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:22.671 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:22.672 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:22.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 1 -- 172.21.15.194:0/3393455700 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 197332543 0 0) 0x7ff604140d50 con 0x7ff604144ec0 2024-03-29T11:24:22.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 1 -- 172.21.15.194:0/3393455700 >> v1:172.21.15.135:6790/0 conn(0x7ff6041454c0 legacy=0x7ff6041447b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:22.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 1 -- 172.21.15.194:0/3393455700 >> v1:172.21.15.135:6789/0 conn(0x7ff60409ac40 legacy=0x7ff6041dad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:22.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:22.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 1 monclient: found mon.b 2024-03-29T11:24:22.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:22.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:22.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 1 -- 172.21.15.194:0/3393455700 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6040b0d30 con 0x7ff604144ec0 2024-03-29T11:24:22.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient: _finish_auth 0 2024-03-29T11:24:22.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:22.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:22.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff60a16b640 5 monclient: authenticate success, global_id 4114 2024-03-29T11:24:22.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff60a16b640 10 monclient: _renew_subs 2024-03-29T11:24:22.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff60a16b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:22.673 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff60a16b640 1 -- 172.21.15.194:0/3393455700 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6040b0f80 con 0x7ff604144ec0 2024-03-29T11:24:22.673 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff60a16b640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:22.673 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff60a16b640 10 monclient: _renew_subs 2024-03-29T11:24:22.673 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff60a16b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:22.673 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff60a16b640 1 -- 172.21.15.194:0/3393455700 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6041e4000 con 0x7ff604144ec0 2024-03-29T11:24:22.673 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff60a16b640 1 librados: init done 2024-03-29T11:24:22.673 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 1 -- 172.21.15.194:0/3393455700 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5fc004e10 con 0x7ff604144ec0 2024-03-29T11:24:22.673 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:22.673 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 1 -- 172.21.15.194:0/3393455700 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7ff5fc005120 con 0x7ff604144ec0 2024-03-29T11:24:22.673 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:22.673 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:22.673 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 monclient: dump: 2024-03-29T11:24:22.673 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:22.673 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:22.673 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:22.674 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:22.674 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:22.674 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:22.674 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:22.674 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:22.674 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:22.674 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:22.674 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff601ffb640 4 set_mon_vals no callback set 2024-03-29T11:24:22.674 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 1 -- 172.21.15.194:0/3393455700 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff5fc005610 con 0x7ff604144ec0 2024-03-29T11:24:22.674 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 1 -- 172.21.15.194:0/3393455700 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 1679602120 0 0) 0x7ff5fc006ae0 con 0x7ff604144ec0 2024-03-29T11:24:22.674 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 10 client.4114.objecter ms_dispatch 0x7ff604000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:24:22.674 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 3 client.4114.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:24:22.674 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 3 client.4114.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:24:22.674 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.673+0000 7ff5ebfff640 20 client.4114.objecter dump_active .. 0 homeless 2024-03-29T11:24:22.674 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.674+0000 7ff60a16b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:22.675 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.674+0000 7ff60a16b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:22.675 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.674+0000 7ff60a16b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:22.675 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.674+0000 7ff60a16b640 1 -- 172.21.15.194:0/3393455700 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5e4005080 con 0x7ff604144ec0 2024-03-29T11:24:22.675 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.678+0000 7ff5ebfff640 1 -- 172.21.15.194:0/3393455700 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff5fc005b00 con 0x7ff604144ec0 2024-03-29T11:24:22.675 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.678+0000 7ff5ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:22.675 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.678+0000 7ff5ebfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:22.732 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.750+0000 7ff60a16b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d8d8500d-c94b-47b0-82f0-8e2c681dd656", "id": 0}] 2024-03-29T11:24:22.732 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.750+0000 7ff60a16b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d8d8500d-c94b-47b0-82f0-8e2c681dd656", "id": 0}] 2024-03-29T11:24:22.733 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.750+0000 7ff60a16b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:22.733 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.750+0000 7ff60a16b640 1 -- 172.21.15.194:0/3393455700 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d8d8500d-c94b-47b0-82f0-8e2c681dd656", "id": 0} v 0) v1 -- 0x7ff5e40058b0 con 0x7ff604144ec0 2024-03-29T11:24:22.738 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.757+0000 7ff5ebfff640 1 -- 172.21.15.194:0/3393455700 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d8d8500d-c94b-47b0-82f0-8e2c681dd656", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 4072105036 0 732707515) 0x7ff5fc005d70 con 0x7ff604144ec0 2024-03-29T11:24:22.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.757+0000 7ff5ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d8d8500d-c94b-47b0-82f0-8e2c681dd656", "id": 0}] 2024-03-29T11:24:22.739 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.757+0000 7ff5ebfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:22.739 INFO:teuthology.orchestra.run.smithi194.stdout:0 2024-03-29T11:24:22.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.757+0000 7ff60a16b640 10 librados: watch_flush enter 2024-03-29T11:24:22.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.757+0000 7ff60a16b640 10 librados: watch_flush exit 2024-03-29T11:24:22.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.757+0000 7ff60a16b640 20 client.4114.objecter shutdown clearing up homeless session... 2024-03-29T11:24:22.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.758+0000 7ff60a16b640 10 client.4114.objecter successfully canceled tick 2024-03-29T11:24:22.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.758+0000 7ff60a16b640 10 monclient: shutdown 2024-03-29T11:24:22.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.758+0000 7ff60a16b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:22.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.758+0000 7ff60a16b640 1 -- 172.21.15.194:0/3393455700 >> v1:172.21.15.194:6789/0 conn(0x7ff604144ec0 legacy=0x7ff6040b0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:22.740 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.758+0000 7ff60a16b640 1 -- 172.21.15.194:0/3393455700 shutdown_connections 2024-03-29T11:24:22.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.758+0000 7ff60a16b640 1 -- 172.21.15.194:0/3393455700 >> 172.21.15.194:0/3393455700 conn(0x7ff604066b00 msgr2=0x7ff60413d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:22.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.758+0000 7ff60a16b640 1 -- 172.21.15.194:0/3393455700 shutdown_connections 2024-03-29T11:24:22.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.758+0000 7ff60a16b640 1 -- 172.21.15.194:0/3393455700 wait complete. 2024-03-29T11:24:22.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.758+0000 7ff60a16b640 1 librados: shutdown 2024-03-29T11:24:22.750 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new bb242707-53a0-44df-88b2-357d7b29d7ff 1 2024-03-29T11:24:22.848 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.866+0000 7f7399740640 10 monclient: get_monmap_and_config 2024-03-29T11:24:22.848 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.866+0000 7f7399740640 10 monclient: build_initial_monmap 2024-03-29T11:24:22.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.866+0000 7f7399740640 10 monclient: monmap: 2024-03-29T11:24:22.849 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-29T11:24:22.849 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:22.849 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:24:22.867921+0000 2024-03-29T11:24:22.849 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:24:22.867921+0000 2024-03-29T11:24:22.849 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:22.849 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:22.849 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:22.849 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:24:22.849 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:24:22.849 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:22.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.867+0000 7f7399740640 1 Processor -- start 2024-03-29T11:24:22.850 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.867+0000 7f7399740640 1 -- start start 2024-03-29T11:24:22.850 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.867+0000 7f7399740640 10 monclient: init 2024-03-29T11:24:22.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.867+0000 7f7399740640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:22.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.867+0000 7f7399740640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:22.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.867+0000 7f7399740640 10 monclient(hunting): picked mon.noname-a con 0x7f73940b8240 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:22.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.867+0000 7f7399740640 10 monclient(hunting): picked mon.noname-b con 0x7f73940b1f60 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:22.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.867+0000 7f7399740640 10 monclient(hunting): picked mon.noname-c con 0x7f739409c8c0 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:22.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.867+0000 7f7399740640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f739409bd80 con 0x7f73940b8240 2024-03-29T11:24:22.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.867+0000 7f7399740640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f739409b960 con 0x7f739409c8c0 2024-03-29T11:24:22.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.867+0000 7f7399740640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73940b2ec0 con 0x7f73940b1f60 2024-03-29T11:24:22.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.867+0000 7f7399740640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:22.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.867+0000 7f7399740640 10 monclient(hunting): authenticate will time out at 770.153834s 2024-03-29T11:24:22.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.867+0000 7f73937fe640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f739409c8c0 0x7f739414d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:57282/0 (socket says 172.21.15.194:57282) 2024-03-29T11:24:22.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.867+0000 7f73937fe640 1 -- 172.21.15.194:0/1127069765 learned_addr learned my addr 172.21.15.194:0/1127069765 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-29T11:24:22.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 1 -- 172.21.15.194:0/1127069765 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3243408783 0 0) 0x7f739409b960 con 0x7f739409c8c0 2024-03-29T11:24:22.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:22.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:22.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:22.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:24:22.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 1 -- 172.21.15.194:0/1127069765 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f737c003520 con 0x7f739409c8c0 2024-03-29T11:24:22.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 1 -- 172.21.15.194:0/1127069765 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 177375011 0 0) 0x7f737c003520 con 0x7f739409c8c0 2024-03-29T11:24:22.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 1 -- 172.21.15.194:0/1127069765 >> v1:172.21.15.135:6790/0 conn(0x7f73940b1f60 legacy=0x7f739409c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:22.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 1 -- 172.21.15.194:0/1127069765 >> v1:172.21.15.135:6789/0 conn(0x7f73940b8240 legacy=0x7f739413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:22.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:24:22.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 1 monclient: found mon.noname-c 2024-03-29T11:24:22.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:22.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:24:22.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 1 -- 172.21.15.194:0/1127069765 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f739413fb60 con 0x7f739409c8c0 2024-03-29T11:24:22.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 10 monclient: _finish_auth 0 2024-03-29T11:24:22.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:24:22.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:22.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 1 -- 172.21.15.194:0/1127069765 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f738c001a10 con 0x7f739409c8c0 2024-03-29T11:24:22.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:22.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:22.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 10 monclient: dump: 2024-03-29T11:24:22.861 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:22.861 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:22.861 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:22.861 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:22.862 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:22.862 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:22.862 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:22.862 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:22.862 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:22.862 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:22.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7399740640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:24:22.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7399740640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:22.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 1 -- 172.21.15.194:0/1127069765 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f738c001b70 con 0x7f739409c8c0 2024-03-29T11:24:22.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:22.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 1 -- 172.21.15.194:0/1127069765 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f738c001fc0 con 0x7f739409c8c0 2024-03-29T11:24:22.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7399740640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:22.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7399740640 4 set_mon_vals no callback set 2024-03-29T11:24:22.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:22.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:22.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7391ffb640 10 monclient: dump: 2024-03-29T11:24:22.863 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:22.863 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:22.863 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:22.863 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:22.863 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:22.863 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:22.863 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:22.863 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:22.863 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:22.864 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:22.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7399740640 10 monclient: shutdown 2024-03-29T11:24:22.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7399740640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:22.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.868+0000 7f7399740640 1 -- 172.21.15.194:0/1127069765 >> v1:172.21.15.194:6789/0 conn(0x7f739409c8c0 legacy=0x7f739414d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:22.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.869+0000 7f7399740640 1 -- 172.21.15.194:0/1127069765 shutdown_connections 2024-03-29T11:24:22.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.869+0000 7f7399740640 1 -- 172.21.15.194:0/1127069765 >> 172.21.15.194:0/1127069765 conn(0x7f7394066b00 msgr2=0x7f7394066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:22.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.869+0000 7f7399740640 1 -- 172.21.15.194:0/1127069765 shutdown_connections 2024-03-29T11:24:22.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.869+0000 7f7399740640 1 -- 172.21.15.194:0/1127069765 wait complete. 2024-03-29T11:24:22.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.869+0000 7f7399740640 10 monclient: build_initial_monmap 2024-03-29T11:24:22.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.869+0000 7f7399740640 10 monclient: monmap: 2024-03-29T11:24:22.864 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-29T11:24:22.864 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:22.864 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 0.000000 2024-03-29T11:24:22.865 INFO:teuthology.orchestra.run.smithi194.stderr:created 0.000000 2024-03-29T11:24:22.865 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:22.865 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:22.865 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:22.865 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:24:22.865 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:24:22.865 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:22.865 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.869+0000 7f7399740640 1 librados: starting msgr at 2024-03-29T11:24:22.865 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.869+0000 7f7399740640 1 librados: starting objecter 2024-03-29T11:24:22.865 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.870+0000 7f7399740640 1 Processor -- start 2024-03-29T11:24:22.865 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.870+0000 7f7399740640 1 -- start start 2024-03-29T11:24:22.865 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.870+0000 7f7399740640 1 librados: setting wanted keys 2024-03-29T11:24:22.865 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.870+0000 7f7399740640 1 librados: calling monclient init 2024-03-29T11:24:22.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.870+0000 7f7399740640 10 monclient: init 2024-03-29T11:24:22.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.870+0000 7f7399740640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:22.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.870+0000 7f7399740640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:22.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.870+0000 7f7399740640 10 monclient(hunting): picked mon.noname-c con 0x7f739409c8c0 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:22.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.870+0000 7f7399740640 10 monclient(hunting): picked mon.noname-b con 0x7f73940b8240 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:22.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.870+0000 7f7399740640 10 monclient(hunting): picked mon.noname-a con 0x7f7394143ee0 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:22.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.870+0000 7f7399740640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7394143850 con 0x7f7394143ee0 2024-03-29T11:24:22.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.870+0000 7f7399740640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7394144400 con 0x7f73940b8240 2024-03-29T11:24:22.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.870+0000 7f7399740640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7394143ac0 con 0x7f739409c8c0 2024-03-29T11:24:22.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.870+0000 7f7399740640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:22.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.870+0000 7f7399740640 10 monclient(hunting): authenticate will time out at 770.156831s 2024-03-29T11:24:22.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.870+0000 7f73927fc640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f73940b8240 0x7f73941323c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:57298/0 (socket says 172.21.15.194:57298) 2024-03-29T11:24:22.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.870+0000 7f73927fc640 1 -- 172.21.15.194:0/779648593 learned_addr learned my addr 172.21.15.194:0/779648593 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-29T11:24:22.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.870+0000 7f737b7fe640 10 client.?.objecter ms_handle_connect 0x7f73940b8240 2024-03-29T11:24:22.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.870+0000 7f737b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:22.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 1 -- 172.21.15.194:0/779648593 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1678614609 0 0) 0x7f7394144400 con 0x7f73940b8240 2024-03-29T11:24:22.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:22.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:22.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:22.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:24:22.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 1 -- 172.21.15.194:0/779648593 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7380004eb0 con 0x7f73940b8240 2024-03-29T11:24:22.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 client.?.objecter ms_handle_connect 0x7f7394143ee0 2024-03-29T11:24:22.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:22.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 client.?.objecter ms_handle_connect 0x7f739409c8c0 2024-03-29T11:24:22.868 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:22.868 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 1 -- 172.21.15.194:0/779648593 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606717172 0 0) 0x7f7394143ac0 con 0x7f739409c8c0 2024-03-29T11:24:22.868 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:22.868 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:22.868 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:22.868 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:24:22.868 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 1 -- 172.21.15.194:0/779648593 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7394144400 con 0x7f739409c8c0 2024-03-29T11:24:22.868 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 1 -- 172.21.15.194:0/779648593 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 176498038 0 0) 0x7f7394143850 con 0x7f7394143ee0 2024-03-29T11:24:22.868 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:22.868 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:22.868 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:22.868 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:24:22.869 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 1 -- 172.21.15.194:0/779648593 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7394143ac0 con 0x7f7394143ee0 2024-03-29T11:24:22.869 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 1 -- 172.21.15.194:0/779648593 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1659262343 0 0) 0x7f7380004eb0 con 0x7f73940b8240 2024-03-29T11:24:22.869 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 1 -- 172.21.15.194:0/779648593 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7394143850 con 0x7f73940b8240 2024-03-29T11:24:22.869 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 1 -- 172.21.15.194:0/779648593 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f7388001920 con 0x7f73940b8240 2024-03-29T11:24:22.869 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:22.869 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:22.869 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient(hunting): dump: 2024-03-29T11:24:22.869 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:22.869 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:22.869 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:22.869 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:22.869 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:22.870 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:22.870 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:22.870 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:22.870 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:22.870 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:22.870 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:22.870 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 1 -- 172.21.15.194:0/779648593 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 661190791 0 0) 0x7f7394144400 con 0x7f739409c8c0 2024-03-29T11:24:22.870 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 1 -- 172.21.15.194:0/779648593 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7380004eb0 con 0x7f739409c8c0 2024-03-29T11:24:22.870 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 1 -- 172.21.15.194:0/779648593 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 197852679 0 0) 0x7f7394143ac0 con 0x7f7394143ee0 2024-03-29T11:24:22.870 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 1 -- 172.21.15.194:0/779648593 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7394144400 con 0x7f7394143ee0 2024-03-29T11:24:22.870 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 1 -- 172.21.15.194:0/779648593 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f7384001790 con 0x7f739409c8c0 2024-03-29T11:24:22.870 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:22.870 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:22.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient(hunting): dump: 2024-03-29T11:24:22.871 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:22.871 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:22.871 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:22.871 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:22.871 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:22.871 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:22.871 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:22.871 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:22.871 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:22.871 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:22.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 1 -- 172.21.15.194:0/779648593 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2376314083 0 0) 0x7f7394143850 con 0x7f73940b8240 2024-03-29T11:24:22.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 1 -- 172.21.15.194:0/779648593 >> v1:172.21.15.135:6790/0 conn(0x7f739409c8c0 legacy=0x7f7394141e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:22.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 1 -- 172.21.15.194:0/779648593 >> v1:172.21.15.135:6789/0 conn(0x7f7394143ee0 legacy=0x7f73941e1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:22.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:22.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 1 monclient: found mon.b 2024-03-29T11:24:22.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:22.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:22.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 1 -- 172.21.15.194:0/779648593 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7394140e30 con 0x7f73940b8240 2024-03-29T11:24:22.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:22.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:22.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:22.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f7399740640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:24:22.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f7399740640 10 monclient: _renew_subs 2024-03-29T11:24:22.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f7399740640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:22.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f7399740640 1 -- 172.21.15.194:0/779648593 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73941414a0 con 0x7f73940b8240 2024-03-29T11:24:22.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f7399740640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:22.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f7399740640 10 monclient: _renew_subs 2024-03-29T11:24:22.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f7399740640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:22.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f7399740640 1 -- 172.21.15.194:0/779648593 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f739413c9f0 con 0x7f73940b8240 2024-03-29T11:24:22.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f7399740640 1 librados: init done 2024-03-29T11:24:22.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 1 -- 172.21.15.194:0/779648593 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7388003610 con 0x7f73940b8240 2024-03-29T11:24:22.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:22.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 1 -- 172.21.15.194:0/779648593 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f7388003800 con 0x7f73940b8240 2024-03-29T11:24:22.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:22.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:22.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f73917fa640 4 set_mon_vals no callback set 2024-03-29T11:24:22.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 monclient: dump: 2024-03-29T11:24:22.874 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:22.874 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:22.874 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:22.874 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:22.874 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:22.874 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:22.874 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:22.874 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:22.874 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:22.874 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:22.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 1 -- 172.21.15.194:0/779648593 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7388003db0 con 0x7f73940b8240 2024-03-29T11:24:22.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 1 -- 172.21.15.194:0/779648593 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 747722337 0 0) 0x7f7388006450 con 0x7f73940b8240 2024-03-29T11:24:22.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 10 client.4120.objecter ms_dispatch 0x7f7394000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:24:22.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 3 client.4120.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:24:22.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.871+0000 7f737b7fe640 3 client.4120.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:24:22.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.872+0000 7f737b7fe640 20 client.4120.objecter dump_active .. 0 homeless 2024-03-29T11:24:22.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.872+0000 7f7399740640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:22.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.872+0000 7f7399740640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:22.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.872+0000 7f7399740640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:22.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.872+0000 7f7399740640 1 -- 172.21.15.194:0/779648593 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7374005080 con 0x7f73940b8240 2024-03-29T11:24:22.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.875+0000 7f737b7fe640 1 -- 172.21.15.194:0/779648593 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7388004240 con 0x7f73940b8240 2024-03-29T11:24:22.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.875+0000 7f737b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:22.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.875+0000 7f737b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:22.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.947+0000 7f7399740640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "bb242707-53a0-44df-88b2-357d7b29d7ff", "id": 1}] 2024-03-29T11:24:22.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.947+0000 7f7399740640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "bb242707-53a0-44df-88b2-357d7b29d7ff", "id": 1}] 2024-03-29T11:24:22.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.947+0000 7f7399740640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:22.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.947+0000 7f7399740640 1 -- 172.21.15.194:0/779648593 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd new", "uuid": "bb242707-53a0-44df-88b2-357d7b29d7ff", "id": 1} v 0) v1 -- 0x7f73740058b0 con 0x7f73940b8240 2024-03-29T11:24:22.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.955+0000 7f737b7fe640 1 -- 172.21.15.194:0/779648593 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bb242707-53a0-44df-88b2-357d7b29d7ff", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2601762632 0 940483276) 0x7f73880057a0 con 0x7f73940b8240 2024-03-29T11:24:22.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.955+0000 7f737b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "bb242707-53a0-44df-88b2-357d7b29d7ff", "id": 1}] 2024-03-29T11:24:22.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.955+0000 7f737b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:22.938 INFO:teuthology.orchestra.run.smithi194.stdout:1 2024-03-29T11:24:22.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.956+0000 7f7399740640 10 librados: watch_flush enter 2024-03-29T11:24:22.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.956+0000 7f7399740640 10 librados: watch_flush exit 2024-03-29T11:24:22.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.956+0000 7f7399740640 20 client.4120.objecter shutdown clearing up homeless session... 2024-03-29T11:24:22.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.956+0000 7f7399740640 10 client.4120.objecter successfully canceled tick 2024-03-29T11:24:22.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.956+0000 7f7399740640 10 monclient: shutdown 2024-03-29T11:24:22.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.956+0000 7f7399740640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:22.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.956+0000 7f7399740640 1 -- 172.21.15.194:0/779648593 >> v1:172.21.15.194:6789/0 conn(0x7f73940b8240 legacy=0x7f73941323c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:22.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.956+0000 7f7399740640 1 -- 172.21.15.194:0/779648593 shutdown_connections 2024-03-29T11:24:22.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.956+0000 7f7399740640 1 -- 172.21.15.194:0/779648593 >> 172.21.15.194:0/779648593 conn(0x7f7394066b00 msgr2=0x7f739413db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:22.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.956+0000 7f7399740640 1 -- 172.21.15.194:0/779648593 shutdown_connections 2024-03-29T11:24:22.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.956+0000 7f7399740640 1 -- 172.21.15.194:0/779648593 wait complete. 2024-03-29T11:24:22.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:22.956+0000 7f7399740640 1 librados: shutdown 2024-03-29T11:24:22.948 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 4e79fc1c-c76e-4d76-bba8-5c18621facf1 2 2024-03-29T11:24:23.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.067+0000 7f4d6bc86640 10 monclient: get_monmap_and_config 2024-03-29T11:24:23.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.067+0000 7f4d6bc86640 10 monclient: build_initial_monmap 2024-03-29T11:24:23.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.067+0000 7f4d6bc86640 10 monclient: monmap: 2024-03-29T11:24:23.049 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-29T11:24:23.050 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.050 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:24:23.069023+0000 2024-03-29T11:24:23.050 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:24:23.069023+0000 2024-03-29T11:24:23.050 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:23.050 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.050 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:23.050 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:24:23.050 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:24:23.050 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.068+0000 7f4d6bc86640 1 Processor -- start 2024-03-29T11:24:23.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.068+0000 7f4d6bc86640 1 -- start start 2024-03-29T11:24:23.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.068+0000 7f4d6bc86640 10 monclient: init 2024-03-29T11:24:23.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.068+0000 7f4d6bc86640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:23.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.068+0000 7f4d6bc86640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:23.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.068+0000 7f4d6bc86640 10 monclient(hunting): picked mon.noname-c con 0x7f4d6409ac40 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:23.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.068+0000 7f4d6bc86640 10 monclient(hunting): picked mon.noname-b con 0x7f4d6409c870 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:23.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.068+0000 7f4d6bc86640 10 monclient(hunting): picked mon.noname-a con 0x7f4d640b4360 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:23.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d6bc86640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d6409dec0 con 0x7f4d640b4360 2024-03-29T11:24:23.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d6bc86640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d6409da30 con 0x7f4d6409ac40 2024-03-29T11:24:23.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d6bc86640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d640b39c0 con 0x7f4d6409c870 2024-03-29T11:24:23.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d6bc86640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:23.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d6bc86640 10 monclient(hunting): authenticate will time out at 770.355116s 2024-03-29T11:24:23.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d69a22640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f4d6409ac40 0x7f4d640b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:57310/0 (socket says 172.21.15.194:57310) 2024-03-29T11:24:23.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d69a22640 1 -- 172.21.15.194:0/3183026069 learned_addr learned my addr 172.21.15.194:0/3183026069 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-29T11:24:23.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 1 -- 172.21.15.194:0/3183026069 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487120991 0 0) 0x7f4d6409da30 con 0x7f4d6409ac40 2024-03-29T11:24:23.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:24:23.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 1 -- 172.21.15.194:0/3183026069 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d50003520 con 0x7f4d6409ac40 2024-03-29T11:24:23.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 1 -- 172.21.15.194:0/3183026069 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1465719315 0 0) 0x7f4d640b39c0 con 0x7f4d6409c870 2024-03-29T11:24:23.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:24:23.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 1 -- 172.21.15.194:0/3183026069 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d6409da30 con 0x7f4d6409c870 2024-03-29T11:24:23.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 1 -- 172.21.15.194:0/3183026069 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440173543 0 0) 0x7f4d5c001490 con 0x7f4d640b4360 2024-03-29T11:24:23.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:24:23.065 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 1 -- 172.21.15.194:0/3183026069 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d640b39c0 con 0x7f4d640b4360 2024-03-29T11:24:23.065 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 1 -- 172.21.15.194:0/3183026069 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 7079479 0 0) 0x7f4d50003520 con 0x7f4d6409ac40 2024-03-29T11:24:23.065 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 1 -- 172.21.15.194:0/3183026069 >> v1:172.21.15.135:6790/0 conn(0x7f4d6409c870 legacy=0x7f4d6413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.065 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 1 -- 172.21.15.194:0/3183026069 >> v1:172.21.15.135:6789/0 conn(0x7f4d640b4360 legacy=0x7f4d6414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.065 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 10 monclient: _finish_hunting 0 2024-03-29T11:24:23.065 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 1 monclient: found mon.noname-c 2024-03-29T11:24:23.065 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:23.065 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:24:23.065 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 1 -- 172.21.15.194:0/3183026069 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d640b5160 con 0x7f4d6409ac40 2024-03-29T11:24:23.065 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 10 monclient: _finish_auth 0 2024-03-29T11:24:23.066 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 10 monclient: _check_auth_tickets 2024-03-29T11:24:23.066 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:23.066 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 1 -- 172.21.15.194:0/3183026069 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f4d60001690 con 0x7f4d6409ac40 2024-03-29T11:24:23.066 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d6bc86640 5 monclient: authenticate success, global_id 4123 2024-03-29T11:24:23.066 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d6bc86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:23.066 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.066 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:23.066 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 10 monclient: dump: 2024-03-29T11:24:23.066 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.066 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.066 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.067 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.067 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.067 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.067 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.067 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.067 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.067 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d6bc86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:23.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 1 -- 172.21.15.194:0/3183026069 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d60001870 con 0x7f4d6409ac40 2024-03-29T11:24:23.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:23.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 1 -- 172.21.15.194:0/3183026069 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f4d60001c80 con 0x7f4d6409ac40 2024-03-29T11:24:23.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d6bc86640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:23.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d6bc86640 4 set_mon_vals no callback set 2024-03-29T11:24:23.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:23.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.069+0000 7f4d68a20640 10 monclient: dump: 2024-03-29T11:24:23.068 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.068 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.068 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.068 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.068 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.068 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.068 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.068 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.068 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.069 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.069 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.070+0000 7f4d6bc86640 10 monclient: shutdown 2024-03-29T11:24:23.069 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.070+0000 7f4d6bc86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:23.069 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.070+0000 7f4d6bc86640 1 -- 172.21.15.194:0/3183026069 >> v1:172.21.15.194:6789/0 conn(0x7f4d6409ac40 legacy=0x7f4d640b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.069 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.070+0000 7f4d6bc86640 1 -- 172.21.15.194:0/3183026069 shutdown_connections 2024-03-29T11:24:23.069 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.070+0000 7f4d6bc86640 1 -- 172.21.15.194:0/3183026069 >> 172.21.15.194:0/3183026069 conn(0x7f4d64066b00 msgr2=0x7f4d64066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:23.069 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.070+0000 7f4d6bc86640 1 -- 172.21.15.194:0/3183026069 shutdown_connections 2024-03-29T11:24:23.069 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.070+0000 7f4d6bc86640 1 -- 172.21.15.194:0/3183026069 wait complete. 2024-03-29T11:24:23.069 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.071+0000 7f4d6bc86640 10 monclient: build_initial_monmap 2024-03-29T11:24:23.069 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.071+0000 7f4d6bc86640 10 monclient: monmap: 2024-03-29T11:24:23.069 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-29T11:24:23.069 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:23.070 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 0.000000 2024-03-29T11:24:23.070 INFO:teuthology.orchestra.run.smithi194.stderr:created 0.000000 2024-03-29T11:24:23.070 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:23.070 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.070 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:23.070 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:24:23.070 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:24:23.070 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.070 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.071+0000 7f4d6bc86640 1 librados: starting msgr at 2024-03-29T11:24:23.070 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.071+0000 7f4d6bc86640 1 librados: starting objecter 2024-03-29T11:24:23.070 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.071+0000 7f4d6bc86640 1 Processor -- start 2024-03-29T11:24:23.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.071+0000 7f4d6bc86640 1 -- start start 2024-03-29T11:24:23.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.071+0000 7f4d6bc86640 1 librados: setting wanted keys 2024-03-29T11:24:23.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.071+0000 7f4d6bc86640 1 librados: calling monclient init 2024-03-29T11:24:23.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.071+0000 7f4d6bc86640 10 monclient: init 2024-03-29T11:24:23.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.071+0000 7f4d6bc86640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:23.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.071+0000 7f4d6bc86640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:23.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.071+0000 7f4d6bc86640 10 monclient(hunting): picked mon.noname-a con 0x7f4d6409ac40 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:23.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d6bc86640 10 monclient(hunting): picked mon.noname-b con 0x7f4d640ae5e0 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:23.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d6bc86640 10 monclient(hunting): picked mon.noname-c con 0x7f4d64136d80 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:23.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d6bc86640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d640af490 con 0x7f4d6409ac40 2024-03-29T11:24:23.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d6bc86640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d640adc90 con 0x7f4d640ae5e0 2024-03-29T11:24:23.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d6bc86640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d64137890 con 0x7f4d64136d80 2024-03-29T11:24:23.072 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d6bc86640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:23.072 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d6bc86640 10 monclient(hunting): authenticate will time out at 770.358138s 2024-03-29T11:24:23.072 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d69221640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f4d640ae5e0 0x7f4d640afe00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:57322/0 (socket says 172.21.15.194:57322) 2024-03-29T11:24:23.072 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d69a22640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f4d6409ac40 0x7f4d640ad580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.194:37726/0 (socket says 172.21.15.194:37726) 2024-03-29T11:24:23.072 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d69221640 1 -- 172.21.15.194:0/2336151334 learned_addr learned my addr 172.21.15.194:0/2336151334 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-29T11:24:23.072 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 client.?.objecter ms_handle_connect 0x7f4d6409ac40 2024-03-29T11:24:23.072 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:23.072 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 client.?.objecter ms_handle_connect 0x7f4d640ae5e0 2024-03-29T11:24:23.072 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:23.072 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 client.?.objecter ms_handle_connect 0x7f4d64136d80 2024-03-29T11:24:23.072 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:23.073 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 1 -- 172.21.15.194:0/2336151334 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1585240021 0 0) 0x7f4d640af490 con 0x7f4d6409ac40 2024-03-29T11:24:23.073 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.073 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.073 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.073 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:24:23.073 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 1 -- 172.21.15.194:0/2336151334 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d48004eb0 con 0x7f4d6409ac40 2024-03-29T11:24:23.073 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 1 -- 172.21.15.194:0/2336151334 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624220941 0 0) 0x7f4d64137890 con 0x7f4d64136d80 2024-03-29T11:24:23.073 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.073 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.073 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.073 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:24:23.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 1 -- 172.21.15.194:0/2336151334 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d640af490 con 0x7f4d64136d80 2024-03-29T11:24:23.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 1 -- 172.21.15.194:0/2336151334 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481541201 0 0) 0x7f4d640adc90 con 0x7f4d640ae5e0 2024-03-29T11:24:23.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:24:23.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 1 -- 172.21.15.194:0/2336151334 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d64137890 con 0x7f4d640ae5e0 2024-03-29T11:24:23.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 1 -- 172.21.15.194:0/2336151334 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1965258728 0 0) 0x7f4d48004eb0 con 0x7f4d6409ac40 2024-03-29T11:24:23.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 1 -- 172.21.15.194:0/2336151334 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d640adc90 con 0x7f4d6409ac40 2024-03-29T11:24:23.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 1 -- 172.21.15.194:0/2336151334 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f4d60001b80 con 0x7f4d6409ac40 2024-03-29T11:24:23.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:23.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 monclient(hunting): dump: 2024-03-29T11:24:23.075 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.075 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.075 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.075 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.075 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.075 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.075 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.075 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.075 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.075 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:23.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 1 -- 172.21.15.194:0/2336151334 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2741228331 0 0) 0x7f4d64137890 con 0x7f4d640ae5e0 2024-03-29T11:24:23.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 1 -- 172.21.15.194:0/2336151334 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d48004eb0 con 0x7f4d640ae5e0 2024-03-29T11:24:23.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 1 -- 172.21.15.194:0/2336151334 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4119959930 0 0) 0x7f4d640af490 con 0x7f4d64136d80 2024-03-29T11:24:23.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 1 -- 172.21.15.194:0/2336151334 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d64137890 con 0x7f4d64136d80 2024-03-29T11:24:23.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 1 -- 172.21.15.194:0/2336151334 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f4d540018b0 con 0x7f4d640ae5e0 2024-03-29T11:24:23.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:23.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 monclient(hunting): dump: 2024-03-29T11:24:23.076 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.076 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.076 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.076 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.077 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.077 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.077 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.077 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.077 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.077 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 1 -- 172.21.15.194:0/2336151334 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f4d5c001b40 con 0x7f4d64136d80 2024-03-29T11:24:23.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:23.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 monclient(hunting): dump: 2024-03-29T11:24:23.077 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.077 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.078 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.078 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.078 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.078 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.078 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.078 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.078 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.078 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.078 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 1 -- 172.21.15.194:0/2336151334 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1666223604 0 0) 0x7f4d640adc90 con 0x7f4d6409ac40 2024-03-29T11:24:23.078 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 1 -- 172.21.15.194:0/2336151334 >> v1:172.21.15.135:6790/0 conn(0x7f4d64136d80 legacy=0x7f4d64137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.078 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 1 -- 172.21.15.194:0/2336151334 >> v1:172.21.15.194:6789/0 conn(0x7f4d640ae5e0 legacy=0x7f4d640afe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:23.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 1 monclient: found mon.a 2024-03-29T11:24:23.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:23.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:23.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 1 -- 172.21.15.194:0/2336151334 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d640af700 con 0x7f4d6409ac40 2024-03-29T11:24:23.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:23.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:23.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:23.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d6bc86640 5 monclient: authenticate success, global_id 4128 2024-03-29T11:24:23.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d6bc86640 10 monclient: _renew_subs 2024-03-29T11:24:23.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d6bc86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:23.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d6bc86640 1 -- 172.21.15.194:0/2336151334 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d640af9e0 con 0x7f4d6409ac40 2024-03-29T11:24:23.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d6bc86640 10 client.4128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:23.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d6bc86640 10 monclient: _renew_subs 2024-03-29T11:24:23.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d6bc86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:23.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d6bc86640 1 -- 172.21.15.194:0/2336151334 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d640b0510 con 0x7f4d6409ac40 2024-03-29T11:24:23.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d6bc86640 1 librados: init done 2024-03-29T11:24:23.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.072+0000 7f4d5a7fc640 1 -- 172.21.15.194:0/2336151334 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d60004bf0 con 0x7f4d6409ac40 2024-03-29T11:24:23.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.073+0000 7f4d5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:23.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.073+0000 7f4d5a7fc640 1 -- 172.21.15.194:0/2336151334 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f4d60004f00 con 0x7f4d6409ac40 2024-03-29T11:24:23.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.073+0000 7f4d5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.073+0000 7f4d5a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:23.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.073+0000 7f4d5a7fc640 10 monclient: dump: 2024-03-29T11:24:23.081 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.081 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.081 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.081 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.081 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.081 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.081 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.081 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.081 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.081 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.073+0000 7f4d5bfff640 4 set_mon_vals no callback set 2024-03-29T11:24:23.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.073+0000 7f4d5a7fc640 1 -- 172.21.15.194:0/2336151334 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4d600053f0 con 0x7f4d6409ac40 2024-03-29T11:24:23.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.073+0000 7f4d5a7fc640 1 -- 172.21.15.194:0/2336151334 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 1031207658 0 0) 0x7f4d60006900 con 0x7f4d6409ac40 2024-03-29T11:24:23.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.073+0000 7f4d5a7fc640 10 client.4128.objecter ms_dispatch 0x7f4d64000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:24:23.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.073+0000 7f4d5a7fc640 3 client.4128.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:24:23.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.073+0000 7f4d5a7fc640 3 client.4128.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:24:23.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.073+0000 7f4d5a7fc640 20 client.4128.objecter dump_active .. 0 homeless 2024-03-29T11:24:23.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.073+0000 7f4d6bc86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.073+0000 7f4d6bc86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.073+0000 7f4d6bc86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:23.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.073+0000 7f4d6bc86640 1 -- 172.21.15.194:0/2336151334 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d4c005080 con 0x7f4d6409ac40 2024-03-29T11:24:23.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.075+0000 7f4d5a7fc640 1 -- 172.21.15.194:0/2336151334 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4d60006590 con 0x7f4d6409ac40 2024-03-29T11:24:23.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.075+0000 7f4d5a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.075+0000 7f4d5a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:23.098 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:23.116+0000 7f992383e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:24:23.112 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-29T11:24:23.130+0000 7fe5b562c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:24:23.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.148+0000 7f4d6bc86640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4e79fc1c-c76e-4d76-bba8-5c18621facf1", "id": 2}] 2024-03-29T11:24:23.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.148+0000 7f4d6bc86640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4e79fc1c-c76e-4d76-bba8-5c18621facf1", "id": 2}] 2024-03-29T11:24:23.131 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.148+0000 7f4d6bc86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:23.131 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.148+0000 7f4d6bc86640 1 -- 172.21.15.194:0/2336151334 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd new", "uuid": "4e79fc1c-c76e-4d76-bba8-5c18621facf1", "id": 2} v 0) v1 -- 0x7f4d4c0058b0 con 0x7f4d6409ac40 2024-03-29T11:24:23.132 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.150+0000 7f4d5a7fc640 1 -- 172.21.15.194:0/2336151334 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4e79fc1c-c76e-4d76-bba8-5c18621facf1", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 4168602321 0 216599125) 0x7f4d60006b90 con 0x7f4d6409ac40 2024-03-29T11:24:23.132 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.150+0000 7f4d5a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4e79fc1c-c76e-4d76-bba8-5c18621facf1", "id": 2}] 2024-03-29T11:24:23.132 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.150+0000 7f4d5a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:23.132 INFO:teuthology.orchestra.run.smithi194.stdout:2 2024-03-29T11:24:23.133 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.151+0000 7f4d6bc86640 10 librados: watch_flush enter 2024-03-29T11:24:23.134 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.151+0000 7f4d6bc86640 10 librados: watch_flush exit 2024-03-29T11:24:23.134 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.151+0000 7f4d6bc86640 20 client.4128.objecter shutdown clearing up homeless session... 2024-03-29T11:24:23.134 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.151+0000 7f4d6bc86640 10 client.4128.objecter successfully canceled tick 2024-03-29T11:24:23.134 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.151+0000 7f4d6bc86640 10 monclient: shutdown 2024-03-29T11:24:23.134 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.151+0000 7f4d6bc86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:23.134 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.151+0000 7f4d6bc86640 1 -- 172.21.15.194:0/2336151334 >> v1:172.21.15.135:6789/0 conn(0x7f4d6409ac40 legacy=0x7f4d640ad580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.134 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.151+0000 7f4d6bc86640 1 -- 172.21.15.194:0/2336151334 shutdown_connections 2024-03-29T11:24:23.134 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.151+0000 7f4d6bc86640 1 -- 172.21.15.194:0/2336151334 >> 172.21.15.194:0/2336151334 conn(0x7f4d64066b00 msgr2=0x7f4d640b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:23.134 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.151+0000 7f4d6bc86640 1 -- 172.21.15.194:0/2336151334 shutdown_connections 2024-03-29T11:24:23.134 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.151+0000 7f4d6bc86640 1 -- 172.21.15.194:0/2336151334 wait complete. 2024-03-29T11:24:23.135 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.151+0000 7f4d6bc86640 1 librados: shutdown 2024-03-29T11:24:23.143 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 335b2809-7389-4a89-b0f2-cfb707d4dd75 3 2024-03-29T11:24:23.153 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:23.171+0000 7f992383e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:24:23.170 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-29T11:24:23.188+0000 7fe5b562c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:24:23.198 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:23.216+0000 7f992383e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:24:23.217 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-29T11:24:23.235+0000 7fe5b562c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:24:23.243 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.261+0000 7f0a358d6640 10 monclient: get_monmap_and_config 2024-03-29T11:24:23.244 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.261+0000 7f0a358d6640 10 monclient: build_initial_monmap 2024-03-29T11:24:23.244 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.261+0000 7f0a358d6640 10 monclient: monmap: 2024-03-29T11:24:23.244 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-29T11:24:23.244 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.244 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:24:23.263226+0000 2024-03-29T11:24:23.244 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:24:23.263226+0000 2024-03-29T11:24:23.244 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:23.244 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.244 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:23.244 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:24:23.244 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:24:23.244 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.245 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.262+0000 7f0a358d6640 1 Processor -- start 2024-03-29T11:24:23.245 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.262+0000 7f0a358d6640 1 -- start start 2024-03-29T11:24:23.245 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.262+0000 7f0a358d6640 10 monclient: init 2024-03-29T11:24:23.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.262+0000 7f0a358d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:23.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.262+0000 7f0a358d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a358d6640 10 monclient(hunting): picked mon.noname-a con 0x7f0a3009ac40 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a358d6640 10 monclient(hunting): picked mon.noname-b con 0x7f0a3009c870 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a358d6640 10 monclient(hunting): picked mon.noname-c con 0x7f0a300b4360 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a358d6640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a3009dec0 con 0x7f0a3009ac40 2024-03-29T11:24:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a358d6640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a3009da30 con 0x7f0a300b4360 2024-03-29T11:24:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a358d6640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a300b39c0 con 0x7f0a3009c870 2024-03-29T11:24:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a358d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a358d6640 10 monclient(hunting): authenticate will time out at 770.549168s 2024-03-29T11:24:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2f7fe640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f0a300b4360 0x7f0a3014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:57330/0 (socket says 172.21.15.194:57330) 2024-03-29T11:24:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2f7fe640 1 -- 172.21.15.194:0/1389899586 learned_addr learned my addr 172.21.15.194:0/1389899586 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-29T11:24:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 1 -- 172.21.15.194:0/1389899586 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583066905 0 0) 0x7f0a3009da30 con 0x7f0a300b4360 2024-03-29T11:24:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:24:23.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 1 -- 172.21.15.194:0/1389899586 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a1c003520 con 0x7f0a300b4360 2024-03-29T11:24:23.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 1 -- 172.21.15.194:0/1389899586 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 131690234 0 0) 0x7f0a18001110 con 0x7f0a3009ac40 2024-03-29T11:24:23.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:24:23.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 1 -- 172.21.15.194:0/1389899586 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a3009da30 con 0x7f0a3009ac40 2024-03-29T11:24:23.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 1 -- 172.21.15.194:0/1389899586 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1113181992 0 0) 0x7f0a300b39c0 con 0x7f0a3009c870 2024-03-29T11:24:23.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:24:23.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 1 -- 172.21.15.194:0/1389899586 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a18001110 con 0x7f0a3009c870 2024-03-29T11:24:23.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 1 -- 172.21.15.194:0/1389899586 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3852131860 0 0) 0x7f0a3009da30 con 0x7f0a3009ac40 2024-03-29T11:24:23.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 1 -- 172.21.15.194:0/1389899586 >> v1:172.21.15.135:6790/0 conn(0x7f0a3009c870 legacy=0x7f0a3013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 1 -- 172.21.15.194:0/1389899586 >> v1:172.21.15.194:6789/0 conn(0x7f0a300b4360 legacy=0x7f0a3014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:24:23.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 1 monclient: found mon.noname-a 2024-03-29T11:24:23.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:23.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.135:6789/0 2024-03-29T11:24:23.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 1 -- 172.21.15.194:0/1389899586 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a300b5160 con 0x7f0a3009ac40 2024-03-29T11:24:23.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 10 monclient: _finish_auth 0 2024-03-29T11:24:23.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:24:23.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:23.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a358d6640 5 monclient: authenticate success, global_id 4131 2024-03-29T11:24:23.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 1 -- 172.21.15.194:0/1389899586 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f0a18001660 con 0x7f0a3009ac40 2024-03-29T11:24:23.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a358d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:23.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:23.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 10 monclient: dump: 2024-03-29T11:24:23.258 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.258 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.258 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.258 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.258 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.258 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.258 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.258 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.258 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.258 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a358d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:23.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 1 -- 172.21.15.194:0/1389899586 <== mon.0 v1:172.21.15.135:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a180017c0 con 0x7f0a3009ac40 2024-03-29T11:24:23.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:23.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a358d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:23.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 1 -- 172.21.15.194:0/1389899586 <== mon.0 v1:172.21.15.135:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f0a18001c10 con 0x7f0a3009ac40 2024-03-29T11:24:23.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a358d6640 4 set_mon_vals no callback set 2024-03-29T11:24:23.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:23.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.263+0000 7f0a2dffb640 10 monclient: dump: 2024-03-29T11:24:23.259 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.259 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.259 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.259 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.259 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.259 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.259 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.259 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.260 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.260 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.264+0000 7f0a358d6640 10 monclient: shutdown 2024-03-29T11:24:23.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.264+0000 7f0a358d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:23.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.264+0000 7f0a358d6640 1 -- 172.21.15.194:0/1389899586 >> v1:172.21.15.135:6789/0 conn(0x7f0a3009ac40 legacy=0x7f0a300b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.264+0000 7f0a358d6640 1 -- 172.21.15.194:0/1389899586 shutdown_connections 2024-03-29T11:24:23.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.264+0000 7f0a358d6640 1 -- 172.21.15.194:0/1389899586 >> 172.21.15.194:0/1389899586 conn(0x7f0a30066b00 msgr2=0x7f0a30066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:23.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.264+0000 7f0a358d6640 1 -- 172.21.15.194:0/1389899586 shutdown_connections 2024-03-29T11:24:23.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.264+0000 7f0a358d6640 1 -- 172.21.15.194:0/1389899586 wait complete. 2024-03-29T11:24:23.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.265+0000 7f0a358d6640 10 monclient: build_initial_monmap 2024-03-29T11:24:23.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.265+0000 7f0a358d6640 10 monclient: monmap: 2024-03-29T11:24:23.260 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-29T11:24:23.260 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:23.260 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 0.000000 2024-03-29T11:24:23.261 INFO:teuthology.orchestra.run.smithi194.stderr:created 0.000000 2024-03-29T11:24:23.261 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:23.261 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.261 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:23.261 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:24:23.261 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:24:23.261 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.265+0000 7f0a358d6640 1 librados: starting msgr at 2024-03-29T11:24:23.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.265+0000 7f0a358d6640 1 librados: starting objecter 2024-03-29T11:24:23.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.265+0000 7f0a358d6640 1 Processor -- start 2024-03-29T11:24:23.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.265+0000 7f0a358d6640 1 -- start start 2024-03-29T11:24:23.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.265+0000 7f0a358d6640 1 librados: setting wanted keys 2024-03-29T11:24:23.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.265+0000 7f0a358d6640 1 librados: calling monclient init 2024-03-29T11:24:23.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.265+0000 7f0a358d6640 10 monclient: init 2024-03-29T11:24:23.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a358d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:23.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a358d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:23.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a358d6640 10 monclient(hunting): picked mon.noname-b con 0x7f0a3009ac40 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:23.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a358d6640 10 monclient(hunting): picked mon.noname-c con 0x7f0a30140400 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:23.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a358d6640 10 monclient(hunting): picked mon.noname-a con 0x7f0a30145950 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:23.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a358d6640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a300b0e90 con 0x7f0a30145950 2024-03-29T11:24:23.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a358d6640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a30145da0 con 0x7f0a3009ac40 2024-03-29T11:24:23.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a358d6640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a30146040 con 0x7f0a30140400 2024-03-29T11:24:23.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a358d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:23.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a358d6640 10 monclient(hunting): authenticate will time out at 770.552249s 2024-03-29T11:24:23.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a2effd640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f0a3009ac40 0x7f0a301411a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:57338/0 (socket says 172.21.15.194:57338) 2024-03-29T11:24:23.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a2effd640 1 -- 172.21.15.194:0/1594867339 learned_addr learned my addr 172.21.15.194:0/1594867339 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-29T11:24:23.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 client.?.objecter ms_handle_connect 0x7f0a3009ac40 2024-03-29T11:24:23.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:23.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 client.?.objecter ms_handle_connect 0x7f0a30140400 2024-03-29T11:24:23.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:23.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 client.?.objecter ms_handle_connect 0x7f0a30145950 2024-03-29T11:24:23.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:23.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 1 -- 172.21.15.194:0/1594867339 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 904415219 0 0) 0x7f0a30145da0 con 0x7f0a3009ac40 2024-03-29T11:24:23.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:24:23.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 1 -- 172.21.15.194:0/1594867339 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a10004eb0 con 0x7f0a3009ac40 2024-03-29T11:24:23.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 1 -- 172.21.15.194:0/1594867339 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320025187 0 0) 0x7f0a300b0e90 con 0x7f0a30145950 2024-03-29T11:24:23.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:24:23.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 1 -- 172.21.15.194:0/1594867339 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a30145da0 con 0x7f0a30145950 2024-03-29T11:24:23.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 1 -- 172.21.15.194:0/1594867339 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146935743 0 0) 0x7f0a30146040 con 0x7f0a30140400 2024-03-29T11:24:23.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:24:23.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 1 -- 172.21.15.194:0/1594867339 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a300b0e90 con 0x7f0a30140400 2024-03-29T11:24:23.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 1 -- 172.21.15.194:0/1594867339 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2974531722 0 0) 0x7f0a30145da0 con 0x7f0a30145950 2024-03-29T11:24:23.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 1 -- 172.21.15.194:0/1594867339 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a30146040 con 0x7f0a30145950 2024-03-29T11:24:23.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 1 -- 172.21.15.194:0/1594867339 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f0a28001a90 con 0x7f0a30145950 2024-03-29T11:24:23.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:23.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 monclient(hunting): dump: 2024-03-29T11:24:23.264 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.265 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.265 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.265 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.265 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.265 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.265 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.265 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.265 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.265 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:23.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 1 -- 172.21.15.194:0/1594867339 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2219082125 0 0) 0x7f0a10004eb0 con 0x7f0a3009ac40 2024-03-29T11:24:23.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 1 -- 172.21.15.194:0/1594867339 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a30145da0 con 0x7f0a3009ac40 2024-03-29T11:24:23.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 1 -- 172.21.15.194:0/1594867339 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f0a18003790 con 0x7f0a3009ac40 2024-03-29T11:24:23.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:23.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 monclient(hunting): dump: 2024-03-29T11:24:23.266 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.266 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.266 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.266 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.266 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.266 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.266 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.266 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.266 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.266 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 1 -- 172.21.15.194:0/1594867339 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1525645197 0 0) 0x7f0a300b0e90 con 0x7f0a30140400 2024-03-29T11:24:23.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 1 -- 172.21.15.194:0/1594867339 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a10004eb0 con 0x7f0a30140400 2024-03-29T11:24:23.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 1 -- 172.21.15.194:0/1594867339 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f0a200018f0 con 0x7f0a30140400 2024-03-29T11:24:23.267 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.267 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:23.267 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 monclient(hunting): dump: 2024-03-29T11:24:23.267 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.267 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.267 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.267 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.267 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.267 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.267 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.267 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.267 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.267 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.267 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 1 -- 172.21.15.194:0/1594867339 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 631028993 0 0) 0x7f0a30146040 con 0x7f0a30145950 2024-03-29T11:24:23.268 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 1 -- 172.21.15.194:0/1594867339 >> v1:172.21.15.135:6790/0 conn(0x7f0a30140400 legacy=0x7f0a30145240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.268 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 1 -- 172.21.15.194:0/1594867339 >> v1:172.21.15.194:6789/0 conn(0x7f0a3009ac40 legacy=0x7f0a301411a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.268 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:23.268 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 1 monclient: found mon.a 2024-03-29T11:24:23.268 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:23.268 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:23.268 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 1 -- 172.21.15.194:0/1594867339 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a301418b0 con 0x7f0a30145950 2024-03-29T11:24:23.268 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 monclient: _finish_auth 0 2024-03-29T11:24:23.268 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:23.268 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:23.268 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a358d6640 5 monclient: authenticate success, global_id 4134 2024-03-29T11:24:23.268 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a358d6640 10 monclient: _renew_subs 2024-03-29T11:24:23.268 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a358d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:23.268 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.266+0000 7f0a358d6640 1 -- 172.21.15.194:0/1594867339 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a301da9b0 con 0x7f0a30145950 2024-03-29T11:24:23.268 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.267+0000 7f0a358d6640 10 client.4134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:23.268 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.267+0000 7f0a358d6640 10 monclient: _renew_subs 2024-03-29T11:24:23.269 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.267+0000 7f0a358d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:23.269 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.267+0000 7f0a358d6640 1 -- 172.21.15.194:0/1594867339 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a301e3630 con 0x7f0a30145950 2024-03-29T11:24:23.269 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.267+0000 7f0a358d6640 1 librados: init done 2024-03-29T11:24:23.269 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.267+0000 7f0a27fff640 1 -- 172.21.15.194:0/1594867339 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a28002000 con 0x7f0a30145950 2024-03-29T11:24:23.269 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.267+0000 7f0a27fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:23.269 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.267+0000 7f0a27fff640 1 -- 172.21.15.194:0/1594867339 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f0a28004d00 con 0x7f0a30145950 2024-03-29T11:24:23.269 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.267+0000 7f0a27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.269 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.267+0000 7f0a2d7fa640 4 set_mon_vals no callback set 2024-03-29T11:24:23.269 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.267+0000 7f0a27fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:23.269 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.267+0000 7f0a27fff640 10 monclient: dump: 2024-03-29T11:24:23.269 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.269 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.269 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.269 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.270 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.270 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.270 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.270 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.270 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.270 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.270 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.267+0000 7f0a27fff640 1 -- 172.21.15.194:0/1594867339 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0a280051f0 con 0x7f0a30145950 2024-03-29T11:24:23.270 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.267+0000 7f0a27fff640 1 -- 172.21.15.194:0/1594867339 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 2044827555 0 0) 0x7f0a28005930 con 0x7f0a30145950 2024-03-29T11:24:23.270 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.267+0000 7f0a27fff640 10 client.4134.objecter ms_dispatch 0x7f0a30000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:24:23.270 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.267+0000 7f0a27fff640 3 client.4134.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:24:23.270 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.267+0000 7f0a27fff640 3 client.4134.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:24:23.270 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.267+0000 7f0a27fff640 20 client.4134.objecter dump_active .. 0 homeless 2024-03-29T11:24:23.270 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.268+0000 7f0a358d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.270 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.268+0000 7f0a358d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.271 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.268+0000 7f0a358d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:23.271 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.268+0000 7f0a358d6640 1 -- 172.21.15.194:0/1594867339 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a14005080 con 0x7f0a30145950 2024-03-29T11:24:23.271 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.270+0000 7f0a27fff640 1 -- 172.21.15.194:0/1594867339 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0a280054a0 con 0x7f0a30145950 2024-03-29T11:24:23.271 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.270+0000 7f0a27fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.271 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.270+0000 7f0a27fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:23.271 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:23.270+0000 7f992383e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:24:23.274 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-29T11:24:23.292+0000 7fe5b562c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:24:23.294 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:23.312+0000 7f992383e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:24:23.318 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-29T11:24:23.336+0000 7fe5b562c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:24:23.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.342+0000 7f0a358d6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "335b2809-7389-4a89-b0f2-cfb707d4dd75", "id": 3}] 2024-03-29T11:24:23.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.342+0000 7f0a358d6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "335b2809-7389-4a89-b0f2-cfb707d4dd75", "id": 3}] 2024-03-29T11:24:23.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.342+0000 7f0a358d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:23.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.342+0000 7f0a358d6640 1 -- 172.21.15.194:0/1594867339 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd new", "uuid": "335b2809-7389-4a89-b0f2-cfb707d4dd75", "id": 3} v 0) v1 -- 0x7f0a140058b0 con 0x7f0a30145950 2024-03-29T11:24:23.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.344+0000 7f0a27fff640 1 -- 172.21.15.194:0/1594867339 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "335b2809-7389-4a89-b0f2-cfb707d4dd75", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 2430280897 0 525046306) 0x7f0a28005c30 con 0x7f0a30145950 2024-03-29T11:24:23.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.344+0000 7f0a27fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "335b2809-7389-4a89-b0f2-cfb707d4dd75", "id": 3}] 2024-03-29T11:24:23.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.344+0000 7f0a27fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:23.326 INFO:teuthology.orchestra.run.smithi194.stdout:3 2024-03-29T11:24:23.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.345+0000 7f0a358d6640 10 librados: watch_flush enter 2024-03-29T11:24:23.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.345+0000 7f0a358d6640 10 librados: watch_flush exit 2024-03-29T11:24:23.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.345+0000 7f0a358d6640 20 client.4134.objecter shutdown clearing up homeless session... 2024-03-29T11:24:23.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.345+0000 7f0a358d6640 10 client.4134.objecter successfully canceled tick 2024-03-29T11:24:23.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.345+0000 7f0a358d6640 10 monclient: shutdown 2024-03-29T11:24:23.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.345+0000 7f0a358d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:23.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.345+0000 7f0a358d6640 1 -- 172.21.15.194:0/1594867339 >> v1:172.21.15.135:6789/0 conn(0x7f0a30145950 legacy=0x7f0a301da2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.345+0000 7f0a358d6640 1 -- 172.21.15.194:0/1594867339 shutdown_connections 2024-03-29T11:24:23.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.345+0000 7f0a358d6640 1 -- 172.21.15.194:0/1594867339 >> 172.21.15.194:0/1594867339 conn(0x7f0a30066b00 msgr2=0x7f0a3013cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:23.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.345+0000 7f0a358d6640 1 -- 172.21.15.194:0/1594867339 shutdown_connections 2024-03-29T11:24:23.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.345+0000 7f0a358d6640 1 -- 172.21.15.194:0/1594867339 wait complete. 2024-03-29T11:24:23.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.345+0000 7f0a358d6640 1 librados: shutdown 2024-03-29T11:24:23.337 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 634e5bb7-ef22-4960-b7e3-0fed0bc2ac99 4 2024-03-29T11:24:23.410 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:23.428+0000 7f992383e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:24:23.439 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-29T11:24:23.458+0000 7fe5b562c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:24:23.477 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.495+0000 7f48b30c1640 10 monclient: get_monmap_and_config 2024-03-29T11:24:23.477 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.495+0000 7f48b30c1640 10 monclient: build_initial_monmap 2024-03-29T11:24:23.477 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.495+0000 7f48b30c1640 10 monclient: monmap: 2024-03-29T11:24:23.477 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-29T11:24:23.477 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.477 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:24:23.496887+0000 2024-03-29T11:24:23.477 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:24:23.496887+0000 2024-03-29T11:24:23.478 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:23.478 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.478 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:23.478 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:24:23.478 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:24:23.478 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.484 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:23.498+0000 7f992383e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:24:23.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.496+0000 7f48b30c1640 1 Processor -- start 2024-03-29T11:24:23.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.496+0000 7f48b30c1640 1 -- start start 2024-03-29T11:24:23.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.496+0000 7f48b30c1640 10 monclient: init 2024-03-29T11:24:23.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.496+0000 7f48b30c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:23.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.496+0000 7f48b30c1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:23.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.496+0000 7f48b30c1640 10 monclient(hunting): picked mon.noname-b con 0x7f48ac09ac40 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:23.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.496+0000 7f48b30c1640 10 monclient(hunting): picked mon.noname-c con 0x7f48ac09c870 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:23.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.496+0000 7f48b30c1640 10 monclient(hunting): picked mon.noname-a con 0x7f48ac0b4360 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:23.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.496+0000 7f48b30c1640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ac09dec0 con 0x7f48ac0b4360 2024-03-29T11:24:23.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.496+0000 7f48b30c1640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ac09da30 con 0x7f48ac09c870 2024-03-29T11:24:23.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.496+0000 7f48b30c1640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ac0b39c0 con 0x7f48ac09ac40 2024-03-29T11:24:23.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.496+0000 7f48b30c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:23.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.496+0000 7f48b30c1640 10 monclient(hunting): authenticate will time out at 770.782966s 2024-03-29T11:24:23.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48abfff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f48ac09c870 0x7f48ac13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:57348/0 (socket says 172.21.15.194:57348) 2024-03-29T11:24:23.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48abfff640 1 -- 172.21.15.194:0/841310688 learned_addr learned my addr 172.21.15.194:0/841310688 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-29T11:24:23.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 1 -- 172.21.15.194:0/841310688 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429904327 0 0) 0x7f48ac09da30 con 0x7f48ac09c870 2024-03-29T11:24:23.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:24:23.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 1 -- 172.21.15.194:0/841310688 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f489c003520 con 0x7f48ac09c870 2024-03-29T11:24:23.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 1 -- 172.21.15.194:0/841310688 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624450002 0 0) 0x7f48a4001490 con 0x7f48ac0b4360 2024-03-29T11:24:23.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:24:23.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 1 -- 172.21.15.194:0/841310688 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48ac09da30 con 0x7f48ac0b4360 2024-03-29T11:24:23.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 1 -- 172.21.15.194:0/841310688 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4166114171 0 0) 0x7f48ac0b39c0 con 0x7f48ac09ac40 2024-03-29T11:24:23.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:24:23.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 1 -- 172.21.15.194:0/841310688 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48a4001490 con 0x7f48ac09ac40 2024-03-29T11:24:23.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 1 -- 172.21.15.194:0/841310688 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 107022372 0 0) 0x7f489c003520 con 0x7f48ac09c870 2024-03-29T11:24:23.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 1 -- 172.21.15.194:0/841310688 >> v1:172.21.15.135:6790/0 conn(0x7f48ac09ac40 legacy=0x7f48ac0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 1 -- 172.21.15.194:0/841310688 >> v1:172.21.15.135:6789/0 conn(0x7f48ac0b4360 legacy=0x7f48ac14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:23.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 1 monclient: found mon.noname-c 2024-03-29T11:24:23.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:23.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:24:23.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 1 -- 172.21.15.194:0/841310688 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ac0b5160 con 0x7f48ac09c870 2024-03-29T11:24:23.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:23.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:23.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:23.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 1 -- 172.21.15.194:0/841310688 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f4898001690 con 0x7f48ac09c870 2024-03-29T11:24:23.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48b30c1640 5 monclient: authenticate success, global_id 4135 2024-03-29T11:24:23.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48b30c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:23.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:23.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 10 monclient: dump: 2024-03-29T11:24:23.492 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.492 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.492 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.492 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.492 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.492 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.492 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.492 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.492 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.492 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48b30c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:23.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 1 -- 172.21.15.194:0/841310688 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4898001870 con 0x7f48ac09c870 2024-03-29T11:24:23.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:23.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 1 -- 172.21.15.194:0/841310688 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f4898001c80 con 0x7f48ac09c870 2024-03-29T11:24:23.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48b30c1640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:23.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48b30c1640 4 set_mon_vals no callback set 2024-03-29T11:24:23.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:23.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.497+0000 7f48ab7fe640 10 monclient: dump: 2024-03-29T11:24:23.493 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.493 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.493 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.493 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.493 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.493 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.493 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.493 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.493 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.493 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.498+0000 7f48b30c1640 10 monclient: shutdown 2024-03-29T11:24:23.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.498+0000 7f48b30c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:23.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.498+0000 7f48b30c1640 1 -- 172.21.15.194:0/841310688 >> v1:172.21.15.194:6789/0 conn(0x7f48ac09c870 legacy=0x7f48ac13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.498+0000 7f48b30c1640 1 -- 172.21.15.194:0/841310688 shutdown_connections 2024-03-29T11:24:23.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.498+0000 7f48b30c1640 1 -- 172.21.15.194:0/841310688 >> 172.21.15.194:0/841310688 conn(0x7f48ac066b00 msgr2=0x7f48ac066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:23.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.498+0000 7f48b30c1640 1 -- 172.21.15.194:0/841310688 shutdown_connections 2024-03-29T11:24:23.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.498+0000 7f48b30c1640 1 -- 172.21.15.194:0/841310688 wait complete. 2024-03-29T11:24:23.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.498+0000 7f48b30c1640 10 monclient: build_initial_monmap 2024-03-29T11:24:23.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.498+0000 7f48b30c1640 10 monclient: monmap: 2024-03-29T11:24:23.494 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-29T11:24:23.494 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:23.494 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 0.000000 2024-03-29T11:24:23.494 INFO:teuthology.orchestra.run.smithi194.stderr:created 0.000000 2024-03-29T11:24:23.494 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:23.494 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.495 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:23.495 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:24:23.495 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:24:23.495 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.499+0000 7f48b30c1640 1 librados: starting msgr at 2024-03-29T11:24:23.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.499+0000 7f48b30c1640 1 librados: starting objecter 2024-03-29T11:24:23.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.499+0000 7f48b30c1640 1 Processor -- start 2024-03-29T11:24:23.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.499+0000 7f48b30c1640 1 -- start start 2024-03-29T11:24:23.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.499+0000 7f48b30c1640 1 librados: setting wanted keys 2024-03-29T11:24:23.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.499+0000 7f48b30c1640 1 librados: calling monclient init 2024-03-29T11:24:23.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.499+0000 7f48b30c1640 10 monclient: init 2024-03-29T11:24:23.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.499+0000 7f48b30c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:23.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.499+0000 7f48b30c1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:23.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.499+0000 7f48b30c1640 10 monclient(hunting): picked mon.noname-b con 0x7f48ac09ac40 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:23.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.499+0000 7f48b30c1640 10 monclient(hunting): picked mon.noname-a con 0x7f48ac0b6b00 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:23.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.499+0000 7f48b30c1640 10 monclient(hunting): picked mon.noname-c con 0x7f48ac1406b0 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:23.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.499+0000 7f48b30c1640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ac1d7e20 con 0x7f48ac0b6b00 2024-03-29T11:24:23.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.499+0000 7f48b30c1640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ac140b00 con 0x7f48ac09ac40 2024-03-29T11:24:23.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.499+0000 7f48b30c1640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ac0ad080 con 0x7f48ac1406b0 2024-03-29T11:24:23.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.499+0000 7f48b30c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:23.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.499+0000 7f48b30c1640 10 monclient(hunting): authenticate will time out at 770.786001s 2024-03-29T11:24:23.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.499+0000 7f48b0e5d640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f48ac09ac40 0x7f48ac1d73a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:57354/0 (socket says 172.21.15.194:57354) 2024-03-29T11:24:23.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48b0e5d640 1 -- 172.21.15.194:0/3166569811 learned_addr learned my addr 172.21.15.194:0/3166569811 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-29T11:24:23.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 client.?.objecter ms_handle_connect 0x7f48ac09ac40 2024-03-29T11:24:23.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:23.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 client.?.objecter ms_handle_connect 0x7f48ac0b6b00 2024-03-29T11:24:23.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:23.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 client.?.objecter ms_handle_connect 0x7f48ac1406b0 2024-03-29T11:24:23.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:23.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 1 -- 172.21.15.194:0/3166569811 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547721333 0 0) 0x7f48ac140b00 con 0x7f48ac09ac40 2024-03-29T11:24:23.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:24:23.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 1 -- 172.21.15.194:0/3166569811 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4890004eb0 con 0x7f48ac09ac40 2024-03-29T11:24:23.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 1 -- 172.21.15.194:0/3166569811 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3331605028 0 0) 0x7f48ac1d7e20 con 0x7f48ac0b6b00 2024-03-29T11:24:23.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:24:23.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 1 -- 172.21.15.194:0/3166569811 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48ac140b00 con 0x7f48ac0b6b00 2024-03-29T11:24:23.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 1 -- 172.21.15.194:0/3166569811 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2267527978 0 0) 0x7f48ac0ad080 con 0x7f48ac1406b0 2024-03-29T11:24:23.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:24:23.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 1 -- 172.21.15.194:0/3166569811 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48ac1d7e20 con 0x7f48ac1406b0 2024-03-29T11:24:23.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 1 -- 172.21.15.194:0/3166569811 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1573596025 0 0) 0x7f48ac140b00 con 0x7f48ac0b6b00 2024-03-29T11:24:23.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 1 -- 172.21.15.194:0/3166569811 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48ac0ad080 con 0x7f48ac0b6b00 2024-03-29T11:24:23.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 1 -- 172.21.15.194:0/3166569811 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f4898001b80 con 0x7f48ac0b6b00 2024-03-29T11:24:23.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:23.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient(hunting): dump: 2024-03-29T11:24:23.498 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.498 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.498 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.498 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.498 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.499 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.499 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.499 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.499 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.499 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:23.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 1 -- 172.21.15.194:0/3166569811 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1717035826 0 0) 0x7f4890004eb0 con 0x7f48ac09ac40 2024-03-29T11:24:23.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 1 -- 172.21.15.194:0/3166569811 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48ac140b00 con 0x7f48ac09ac40 2024-03-29T11:24:23.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 1 -- 172.21.15.194:0/3166569811 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f48a0001a10 con 0x7f48ac09ac40 2024-03-29T11:24:23.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:23.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient(hunting): dump: 2024-03-29T11:24:23.499 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.499 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.500 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.500 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.500 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.500 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.500 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.500 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.500 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.500 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.500 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 1 -- 172.21.15.194:0/3166569811 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1682248899 0 0) 0x7f48ac1d7e20 con 0x7f48ac1406b0 2024-03-29T11:24:23.500 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 1 -- 172.21.15.194:0/3166569811 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4890004eb0 con 0x7f48ac1406b0 2024-03-29T11:24:23.500 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 1 -- 172.21.15.194:0/3166569811 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f48a4001a40 con 0x7f48ac1406b0 2024-03-29T11:24:23.500 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.500 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:23.500 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient(hunting): dump: 2024-03-29T11:24:23.501 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.501 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.501 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.501 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.501 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.501 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.501 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.501 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.501 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.501 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 1 -- 172.21.15.194:0/3166569811 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1451884253 0 0) 0x7f48ac0ad080 con 0x7f48ac0b6b00 2024-03-29T11:24:23.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 1 -- 172.21.15.194:0/3166569811 >> v1:172.21.15.135:6790/0 conn(0x7f48ac1406b0 legacy=0x7f48ac1e1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 1 -- 172.21.15.194:0/3166569811 >> v1:172.21.15.194:6789/0 conn(0x7f48ac09ac40 legacy=0x7f48ac1d73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:24:23.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 1 monclient: found mon.a 2024-03-29T11:24:23.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:23.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:23.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 1 -- 172.21.15.194:0/3166569811 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ac1d7ab0 con 0x7f48ac0b6b00 2024-03-29T11:24:23.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient: _finish_auth 0 2024-03-29T11:24:23.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:24:23.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:23.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48b30c1640 5 monclient: authenticate success, global_id 4140 2024-03-29T11:24:23.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48b30c1640 10 monclient: _renew_subs 2024-03-29T11:24:23.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48b30c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:23.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48b30c1640 1 -- 172.21.15.194:0/3166569811 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48ac1e3570 con 0x7f48ac0b6b00 2024-03-29T11:24:23.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48b30c1640 10 client.4140.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:23.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48b30c1640 10 monclient: _renew_subs 2024-03-29T11:24:23.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48b30c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:23.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48b30c1640 1 -- 172.21.15.194:0/3166569811 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f48ac1e38e0 con 0x7f48ac0b6b00 2024-03-29T11:24:23.503 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48b30c1640 1 librados: init done 2024-03-29T11:24:23.503 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 1 -- 172.21.15.194:0/3166569811 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4898004bf0 con 0x7f48ac0b6b00 2024-03-29T11:24:23.503 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:23.503 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 1 -- 172.21.15.194:0/3166569811 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f48980010e0 con 0x7f48ac0b6b00 2024-03-29T11:24:23.503 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.503 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:23.503 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48aaffd640 4 set_mon_vals no callback set 2024-03-29T11:24:23.503 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 monclient: dump: 2024-03-29T11:24:23.503 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.503 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.503 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.503 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.503 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.503 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.504 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.504 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.504 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.504 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 1 -- 172.21.15.194:0/3166569811 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4898005310 con 0x7f48ac0b6b00 2024-03-29T11:24:23.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 1 -- 172.21.15.194:0/3166569811 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2490311442 0 0) 0x7f4898005ad0 con 0x7f48ac0b6b00 2024-03-29T11:24:23.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 10 client.4140.objecter ms_dispatch 0x7f48ac000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:24:23.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 3 client.4140.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:24:23.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 3 client.4140.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:24:23.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.500+0000 7f48a97fa640 20 client.4140.objecter dump_active .. 0 homeless 2024-03-29T11:24:23.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.501+0000 7f48b30c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.501+0000 7f48b30c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.501+0000 7f48b30c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:23.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.501+0000 7f48b30c1640 1 -- 172.21.15.194:0/3166569811 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f488c005080 con 0x7f48ac0b6b00 2024-03-29T11:24:23.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.503+0000 7f48a97fa640 1 -- 172.21.15.194:0/3166569811 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4898005db0 con 0x7f48ac0b6b00 2024-03-29T11:24:23.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.503+0000 7f48a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.503+0000 7f48a97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:23.512 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-29T11:24:23.530+0000 7fe5b562c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:24:23.527 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:23.545+0000 7f992383e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:24:23.555 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.573+0000 7f48b30c1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "634e5bb7-ef22-4960-b7e3-0fed0bc2ac99", "id": 4}] 2024-03-29T11:24:23.555 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.573+0000 7f48b30c1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "634e5bb7-ef22-4960-b7e3-0fed0bc2ac99", "id": 4}] 2024-03-29T11:24:23.555 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.573+0000 7f48b30c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:23.555 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.573+0000 7f48b30c1640 1 -- 172.21.15.194:0/3166569811 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd new", "uuid": "634e5bb7-ef22-4960-b7e3-0fed0bc2ac99", "id": 4} v 0) v1 -- 0x7f488c0058b0 con 0x7f48ac0b6b00 2024-03-29T11:24:23.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.575+0000 7f48a97fa640 1 -- 172.21.15.194:0/3166569811 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "634e5bb7-ef22-4960-b7e3-0fed0bc2ac99", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 1995669829 0 1697012583) 0x7f48980055c0 con 0x7f48ac0b6b00 2024-03-29T11:24:23.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.575+0000 7f48a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "634e5bb7-ef22-4960-b7e3-0fed0bc2ac99", "id": 4}] 2024-03-29T11:24:23.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.575+0000 7f48a97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:23.557 INFO:teuthology.orchestra.run.smithi194.stdout:4 2024-03-29T11:24:23.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.576+0000 7f48b30c1640 10 librados: watch_flush enter 2024-03-29T11:24:23.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.576+0000 7f48b30c1640 10 librados: watch_flush exit 2024-03-29T11:24:23.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.576+0000 7f48b30c1640 20 client.4140.objecter shutdown clearing up homeless session... 2024-03-29T11:24:23.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.576+0000 7f48b30c1640 10 client.4140.objecter successfully canceled tick 2024-03-29T11:24:23.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.576+0000 7f48b30c1640 10 monclient: shutdown 2024-03-29T11:24:23.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.576+0000 7f48b30c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:23.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.576+0000 7f48b30c1640 1 -- 172.21.15.194:0/3166569811 >> v1:172.21.15.135:6789/0 conn(0x7f48ac0b6b00 legacy=0x7f48ac13ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.576+0000 7f48b30c1640 1 -- 172.21.15.194:0/3166569811 shutdown_connections 2024-03-29T11:24:23.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.576+0000 7f48b30c1640 1 -- 172.21.15.194:0/3166569811 >> 172.21.15.194:0/3166569811 conn(0x7f48ac066b00 msgr2=0x7f48ac0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:23.558 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-29T11:24:23.576+0000 7fe5b562c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:24:23.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.576+0000 7f48b30c1640 1 -- 172.21.15.194:0/3166569811 shutdown_connections 2024-03-29T11:24:23.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.576+0000 7f48b30c1640 1 -- 172.21.15.194:0/3166569811 wait complete. 2024-03-29T11:24:23.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.576+0000 7f48b30c1640 1 librados: shutdown 2024-03-29T11:24:23.568 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new bcbd29f0-a4d9-4708-b160-5cd17058942f 5 2024-03-29T11:24:23.580 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:23.599+0000 7f992383e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:24:23.614 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-29T11:24:23.632+0000 7fe5b562c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:24:23.627 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:23.645+0000 7f992383e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:24:23.663 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-29T11:24:23.681+0000 7fe5b562c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:24:23.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.685+0000 7fee10c33640 10 monclient: get_monmap_and_config 2024-03-29T11:24:23.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.685+0000 7fee10c33640 10 monclient: build_initial_monmap 2024-03-29T11:24:23.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.685+0000 7fee10c33640 10 monclient: monmap: 2024-03-29T11:24:23.667 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-29T11:24:23.667 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.667 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:24:23.686942+0000 2024-03-29T11:24:23.667 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:24:23.686942+0000 2024-03-29T11:24:23.667 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:23.667 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.667 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:23.668 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:24:23.668 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:24:23.668 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.676 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.686+0000 7fee10c33640 1 Processor -- start 2024-03-29T11:24:23.676 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.686+0000 7fee10c33640 1 -- start start 2024-03-29T11:24:23.676 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.686+0000 7fee10c33640 10 monclient: init 2024-03-29T11:24:23.676 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.686+0000 7fee10c33640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:23.676 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.686+0000 7fee10c33640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:23.676 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.686+0000 7fee10c33640 10 monclient(hunting): picked mon.noname-b con 0x7fee0c066b90 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:23.676 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.686+0000 7fee10c33640 10 monclient(hunting): picked mon.noname-a con 0x7fee0c13f340 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:23.676 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.686+0000 7fee10c33640 10 monclient(hunting): picked mon.noname-c con 0x7fee0c13f7a0 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:23.676 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.686+0000 7fee10c33640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c13fc30 con 0x7fee0c13f340 2024-03-29T11:24:23.676 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.686+0000 7fee10c33640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c0afd70 con 0x7fee0c13f7a0 2024-03-29T11:24:23.676 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.686+0000 7fee10c33640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c0aec00 con 0x7fee0c066b90 2024-03-29T11:24:23.676 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.686+0000 7fee10c33640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:23.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.686+0000 7fee10c33640 10 monclient(hunting): authenticate will time out at 770.973023s 2024-03-29T11:24:23.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0ad9d640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fee0c13f7a0 0x7fee0c142770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:57364/0 (socket says 172.21.15.194:57364) 2024-03-29T11:24:23.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0ad9d640 1 -- 172.21.15.194:0/3742320245 learned_addr learned my addr 172.21.15.194:0/3742320245 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-29T11:24:23.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 1 -- 172.21.15.194:0/3742320245 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 915277503 0 0) 0x7fee0c0afd70 con 0x7fee0c13f7a0 2024-03-29T11:24:23.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:24:23.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 1 -- 172.21.15.194:0/3742320245 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedf4003520 con 0x7fee0c13f7a0 2024-03-29T11:24:23.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 1 -- 172.21.15.194:0/3742320245 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701934797 0 0) 0x7fee0c13fc30 con 0x7fee0c13f340 2024-03-29T11:24:23.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:24:23.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 1 -- 172.21.15.194:0/3742320245 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee0c0afd70 con 0x7fee0c13f340 2024-03-29T11:24:23.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 1 -- 172.21.15.194:0/3742320245 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311509441 0 0) 0x7fee0c0aec00 con 0x7fee0c066b90 2024-03-29T11:24:23.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:24:23.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 1 -- 172.21.15.194:0/3742320245 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee0c13fc30 con 0x7fee0c066b90 2024-03-29T11:24:23.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 1 -- 172.21.15.194:0/3742320245 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4013051379 0 0) 0x7fedf4003520 con 0x7fee0c13f7a0 2024-03-29T11:24:23.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 1 -- 172.21.15.194:0/3742320245 >> v1:172.21.15.135:6790/0 conn(0x7fee0c066b90 legacy=0x7fee0c0af580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 1 -- 172.21.15.194:0/3742320245 >> v1:172.21.15.135:6789/0 conn(0x7fee0c13f340 legacy=0x7fee0c142060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 10 monclient: _finish_hunting 0 2024-03-29T11:24:23.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 1 monclient: found mon.noname-c 2024-03-29T11:24:23.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:23.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:24:23.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 1 -- 172.21.15.194:0/3742320245 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c0aed90 con 0x7fee0c13f7a0 2024-03-29T11:24:23.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 10 monclient: _finish_auth 0 2024-03-29T11:24:23.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 10 monclient: _check_auth_tickets 2024-03-29T11:24:23.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:23.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 1 -- 172.21.15.194:0/3742320245 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fee04001a10 con 0x7fee0c13f7a0 2024-03-29T11:24:23.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee10c33640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:24:23.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee10c33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:23.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:23.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 10 monclient: dump: 2024-03-29T11:24:23.679 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.679 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.679 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.679 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.679 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.680 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.680 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.680 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.680 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.680 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee10c33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:23.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 1 -- 172.21.15.194:0/3742320245 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee04001bf0 con 0x7fee0c13f7a0 2024-03-29T11:24:23.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:23.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee10c33640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:23.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 1 -- 172.21.15.194:0/3742320245 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fee04002000 con 0x7fee0c13f7a0 2024-03-29T11:24:23.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee10c33640 4 set_mon_vals no callback set 2024-03-29T11:24:23.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:23.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.687+0000 7fee0959a640 10 monclient: dump: 2024-03-29T11:24:23.680 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.681 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.681 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.681 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.681 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.681 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.681 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.681 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.681 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.681 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.688+0000 7fee10c33640 10 monclient: shutdown 2024-03-29T11:24:23.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.688+0000 7fee10c33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:23.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.688+0000 7fee10c33640 1 -- 172.21.15.194:0/3742320245 >> v1:172.21.15.194:6789/0 conn(0x7fee0c13f7a0 legacy=0x7fee0c142770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.688+0000 7fee10c33640 1 -- 172.21.15.194:0/3742320245 shutdown_connections 2024-03-29T11:24:23.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.688+0000 7fee10c33640 1 -- 172.21.15.194:0/3742320245 >> 172.21.15.194:0/3742320245 conn(0x7fee0c0b4970 msgr2=0x7fee0c0b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:23.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.688+0000 7fee10c33640 1 -- 172.21.15.194:0/3742320245 shutdown_connections 2024-03-29T11:24:23.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.688+0000 7fee10c33640 1 -- 172.21.15.194:0/3742320245 wait complete. 2024-03-29T11:24:23.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.688+0000 7fee10c33640 10 monclient: build_initial_monmap 2024-03-29T11:24:23.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.688+0000 7fee10c33640 10 monclient: monmap: 2024-03-29T11:24:23.682 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-29T11:24:23.682 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:23.682 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 0.000000 2024-03-29T11:24:23.682 INFO:teuthology.orchestra.run.smithi194.stderr:created 0.000000 2024-03-29T11:24:23.682 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:23.682 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.682 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:23.682 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:24:23.682 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:24:23.682 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.689+0000 7fee10c33640 1 librados: starting msgr at 2024-03-29T11:24:23.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.689+0000 7fee10c33640 1 librados: starting objecter 2024-03-29T11:24:23.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.689+0000 7fee10c33640 1 Processor -- start 2024-03-29T11:24:23.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.689+0000 7fee10c33640 1 -- start start 2024-03-29T11:24:23.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.689+0000 7fee10c33640 1 librados: setting wanted keys 2024-03-29T11:24:23.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.689+0000 7fee10c33640 1 librados: calling monclient init 2024-03-29T11:24:23.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.689+0000 7fee10c33640 10 monclient: init 2024-03-29T11:24:23.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.689+0000 7fee10c33640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:23.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.689+0000 7fee10c33640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:23.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.689+0000 7fee10c33640 10 monclient(hunting): picked mon.noname-a con 0x7fee0c066b90 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:23.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.689+0000 7fee10c33640 10 monclient(hunting): picked mon.noname-c con 0x7fee0c13f340 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:23.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.689+0000 7fee10c33640 10 monclient(hunting): picked mon.noname-b con 0x7fee0c13f7a0 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:23.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.689+0000 7fee10c33640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c138190 con 0x7fee0c066b90 2024-03-29T11:24:23.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.689+0000 7fee10c33640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c138400 con 0x7fee0c13f7a0 2024-03-29T11:24:23.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.689+0000 7fee10c33640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0c1386e0 con 0x7fee0c13f340 2024-03-29T11:24:23.684 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.689+0000 7fee10c33640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:23.684 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.689+0000 7fee10c33640 10 monclient(hunting): authenticate will time out at 770.975975s 2024-03-29T11:24:23.684 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.689+0000 7fee0ad9d640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fee0c13f7a0 0x7fee0c1e1d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:57368/0 (socket says 172.21.15.194:57368) 2024-03-29T11:24:23.684 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.689+0000 7fee0ad9d640 1 -- 172.21.15.194:0/4180267526 learned_addr learned my addr 172.21.15.194:0/4180267526 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-29T11:24:23.684 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 client.?.objecter ms_handle_connect 0x7fee0c13f7a0 2024-03-29T11:24:23.684 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:23.684 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 client.?.objecter ms_handle_connect 0x7fee0c13f340 2024-03-29T11:24:23.684 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:23.684 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 client.?.objecter ms_handle_connect 0x7fee0c066b90 2024-03-29T11:24:23.684 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:23.684 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 1 -- 172.21.15.194:0/4180267526 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067901922 0 0) 0x7fee0c138400 con 0x7fee0c13f7a0 2024-03-29T11:24:23.684 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.684 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.684 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.684 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:24:23.685 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 1 -- 172.21.15.194:0/4180267526 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedec004eb0 con 0x7fee0c13f7a0 2024-03-29T11:24:23.685 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 1 -- 172.21.15.194:0/4180267526 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505774878 0 0) 0x7fee0c1386e0 con 0x7fee0c13f340 2024-03-29T11:24:23.685 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.685 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.685 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.685 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:24:23.685 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 1 -- 172.21.15.194:0/4180267526 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee0c138400 con 0x7fee0c13f340 2024-03-29T11:24:23.685 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 1 -- 172.21.15.194:0/4180267526 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3454242574 0 0) 0x7fee0c138190 con 0x7fee0c066b90 2024-03-29T11:24:23.685 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.685 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.685 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.685 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:24:23.685 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 1 -- 172.21.15.194:0/4180267526 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee0c1386e0 con 0x7fee0c066b90 2024-03-29T11:24:23.685 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 1 -- 172.21.15.194:0/4180267526 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2025468736 0 0) 0x7fedec004eb0 con 0x7fee0c13f7a0 2024-03-29T11:24:23.685 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 1 -- 172.21.15.194:0/4180267526 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee0c138190 con 0x7fee0c13f7a0 2024-03-29T11:24:23.686 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 1 -- 172.21.15.194:0/4180267526 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fee04001f00 con 0x7fee0c13f7a0 2024-03-29T11:24:23.686 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.686 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:23.686 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient(hunting): dump: 2024-03-29T11:24:23.686 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.686 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.686 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.686 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.686 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.686 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.686 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.686 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.686 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.686 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.686 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:23.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 1 -- 172.21.15.194:0/4180267526 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 165946546 0 0) 0x7fee0c138400 con 0x7fee0c13f340 2024-03-29T11:24:23.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 1 -- 172.21.15.194:0/4180267526 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fedec004eb0 con 0x7fee0c13f340 2024-03-29T11:24:23.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 1 -- 172.21.15.194:0/4180267526 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2329737707 0 0) 0x7fee0c1386e0 con 0x7fee0c066b90 2024-03-29T11:24:23.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 1 -- 172.21.15.194:0/4180267526 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee0c138400 con 0x7fee0c066b90 2024-03-29T11:24:23.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 1 -- 172.21.15.194:0/4180267526 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fee00001930 con 0x7fee0c13f340 2024-03-29T11:24:23.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:23.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient(hunting): dump: 2024-03-29T11:24:23.687 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.687 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.687 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.687 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.687 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.688 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.688 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.688 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.688 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.688 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.688 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 1 -- 172.21.15.194:0/4180267526 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fedfc001790 con 0x7fee0c066b90 2024-03-29T11:24:23.688 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.688 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:23.688 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient(hunting): dump: 2024-03-29T11:24:23.688 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.688 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.688 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.688 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.688 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.688 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.689 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.689 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.689 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.689 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.689 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 1 -- 172.21.15.194:0/4180267526 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2391638591 0 0) 0x7fee0c138190 con 0x7fee0c13f7a0 2024-03-29T11:24:23.689 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 1 -- 172.21.15.194:0/4180267526 >> v1:172.21.15.135:6790/0 conn(0x7fee0c13f340 legacy=0x7fee0c140ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.689 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 1 -- 172.21.15.194:0/4180267526 >> v1:172.21.15.135:6789/0 conn(0x7fee0c066b90 legacy=0x7fee0c1373c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.689 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:23.689 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 1 monclient: found mon.b 2024-03-29T11:24:23.689 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:23.689 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:23.689 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 1 -- 172.21.15.194:0/4180267526 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0c1389c0 con 0x7fee0c13f7a0 2024-03-29T11:24:23.689 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:23.689 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:23.689 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:23.690 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fee10c33640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:24:23.690 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fee10c33640 10 monclient: _renew_subs 2024-03-29T11:24:23.690 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fee10c33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:23.690 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fee10c33640 1 -- 172.21.15.194:0/4180267526 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee0c136ea0 con 0x7fee0c13f7a0 2024-03-29T11:24:23.690 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fee10c33640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:23.690 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fee10c33640 10 monclient: _renew_subs 2024-03-29T11:24:23.690 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fee10c33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:23.690 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fee10c33640 1 -- 172.21.15.194:0/4180267526 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee0c1365d0 con 0x7fee0c13f7a0 2024-03-29T11:24:23.690 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fee10c33640 1 librados: init done 2024-03-29T11:24:23.690 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 1 -- 172.21.15.194:0/4180267526 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee04004f70 con 0x7fee0c13f7a0 2024-03-29T11:24:23.690 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:23.690 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 1 -- 172.21.15.194:0/4180267526 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fee04005280 con 0x7fee0c13f7a0 2024-03-29T11:24:23.690 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.690 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:23.691 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 monclient: dump: 2024-03-29T11:24:23.691 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.691 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.691 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.691 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.691 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.691 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.691 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.691 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.691 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.691 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.691 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fee08d99640 4 set_mon_vals no callback set 2024-03-29T11:24:23.691 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 1 -- 172.21.15.194:0/4180267526 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fee04005770 con 0x7fee0c13f7a0 2024-03-29T11:24:23.691 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 1 -- 172.21.15.194:0/4180267526 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 2728764388 0 0) 0x7fee04005fa0 con 0x7fee0c13f7a0 2024-03-29T11:24:23.691 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 10 client.4144.objecter ms_dispatch 0x7fee0c000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:24:23.692 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:24:23.692 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 3 client.4144.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:24:23.692 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.690+0000 7fedfaffd640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-29T11:24:23.692 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.691+0000 7fee10c33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.692 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.691+0000 7fee10c33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.692 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.691+0000 7fee10c33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:23.692 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.691+0000 7fee10c33640 1 -- 172.21.15.194:0/4180267526 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf0005080 con 0x7fee0c13f7a0 2024-03-29T11:24:23.692 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.693+0000 7fedfaffd640 1 -- 172.21.15.194:0/4180267526 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fee04006200 con 0x7fee0c13f7a0 2024-03-29T11:24:23.692 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.693+0000 7fedfaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.692 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.693+0000 7fedfaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:23.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.765+0000 7fee10c33640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "bcbd29f0-a4d9-4708-b160-5cd17058942f", "id": 5}] 2024-03-29T11:24:23.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.765+0000 7fee10c33640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "bcbd29f0-a4d9-4708-b160-5cd17058942f", "id": 5}] 2024-03-29T11:24:23.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.765+0000 7fee10c33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:23.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.765+0000 7fee10c33640 1 -- 172.21.15.194:0/4180267526 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd new", "uuid": "bcbd29f0-a4d9-4708-b160-5cd17058942f", "id": 5} v 0) v1 -- 0x7fedf00058b0 con 0x7fee0c13f7a0 2024-03-29T11:24:23.750 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.769+0000 7fedfaffd640 1 -- 172.21.15.194:0/4180267526 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bcbd29f0-a4d9-4708-b160-5cd17058942f", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 576823104 0 1988412176) 0x7fee04005a20 con 0x7fee0c13f7a0 2024-03-29T11:24:23.750 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.769+0000 7fedfaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "bcbd29f0-a4d9-4708-b160-5cd17058942f", "id": 5}] 2024-03-29T11:24:23.750 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.769+0000 7fedfaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:23.751 INFO:teuthology.orchestra.run.smithi194.stdout:5 2024-03-29T11:24:23.752 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.769+0000 7fee10c33640 10 librados: watch_flush enter 2024-03-29T11:24:23.752 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.769+0000 7fee10c33640 10 librados: watch_flush exit 2024-03-29T11:24:23.752 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.769+0000 7fee10c33640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-29T11:24:23.752 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.769+0000 7fee10c33640 10 client.4144.objecter successfully canceled tick 2024-03-29T11:24:23.752 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.770+0000 7fee10c33640 10 monclient: shutdown 2024-03-29T11:24:23.752 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.770+0000 7fee10c33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:23.752 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.770+0000 7fee10c33640 1 -- 172.21.15.194:0/4180267526 >> v1:172.21.15.194:6789/0 conn(0x7fee0c13f7a0 legacy=0x7fee0c1e1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.770+0000 7fee10c33640 1 -- 172.21.15.194:0/4180267526 shutdown_connections 2024-03-29T11:24:23.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.770+0000 7fee10c33640 1 -- 172.21.15.194:0/4180267526 >> 172.21.15.194:0/4180267526 conn(0x7fee0c0b4970 msgr2=0x7fee0c0b2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:23.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.770+0000 7fee10c33640 1 -- 172.21.15.194:0/4180267526 shutdown_connections 2024-03-29T11:24:23.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.770+0000 7fee10c33640 1 -- 172.21.15.194:0/4180267526 wait complete. 2024-03-29T11:24:23.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.770+0000 7fee10c33640 1 librados: shutdown 2024-03-29T11:24:23.762 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new b27f71ce-28e4-4daf-b25f-6fe58abcb572 6 2024-03-29T11:24:23.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.879+0000 7f97087af640 10 monclient: get_monmap_and_config 2024-03-29T11:24:23.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.879+0000 7f97087af640 10 monclient: build_initial_monmap 2024-03-29T11:24:23.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.879+0000 7f97087af640 10 monclient: monmap: 2024-03-29T11:24:23.861 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-29T11:24:23.861 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.862 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:24:23.880649+0000 2024-03-29T11:24:23.862 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:24:23.880649+0000 2024-03-29T11:24:23.862 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:23.862 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.862 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:23.862 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:24:23.862 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:24:23.862 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.879+0000 7f97087af640 1 Processor -- start 2024-03-29T11:24:23.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.879+0000 7f97087af640 1 -- start start 2024-03-29T11:24:23.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.879+0000 7f97087af640 10 monclient: init 2024-03-29T11:24:23.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.880+0000 7f97087af640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:23.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.880+0000 7f97087af640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:23.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.880+0000 7f97087af640 10 monclient(hunting): picked mon.noname-a con 0x7f970009ac40 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:23.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.880+0000 7f97087af640 10 monclient(hunting): picked mon.noname-b con 0x7f970009c870 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:23.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.880+0000 7f97087af640 10 monclient(hunting): picked mon.noname-c con 0x7f97000b4360 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:23.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.880+0000 7f97087af640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f970009dec0 con 0x7f970009ac40 2024-03-29T11:24:23.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.880+0000 7f97087af640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f970009da30 con 0x7f97000b4360 2024-03-29T11:24:23.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.880+0000 7f97087af640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97000b39c0 con 0x7f970009c870 2024-03-29T11:24:23.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.880+0000 7f97087af640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:23.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.880+0000 7f97087af640 10 monclient(hunting): authenticate will time out at 771.166816s 2024-03-29T11:24:23.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.880+0000 7f9706d4c640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f97000b4360 0x7f970014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:57374/0 (socket says 172.21.15.194:57374) 2024-03-29T11:24:23.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.880+0000 7f9706d4c640 1 -- 172.21.15.194:0/2362815230 learned_addr learned my addr 172.21.15.194:0/2362815230 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-29T11:24:23.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 1 -- 172.21.15.194:0/2362815230 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498197327 0 0) 0x7f970009da30 con 0x7f97000b4360 2024-03-29T11:24:23.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:24:23.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 1 -- 172.21.15.194:0/2362815230 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96ec003520 con 0x7f97000b4360 2024-03-29T11:24:23.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 1 -- 172.21.15.194:0/2362815230 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1411920157 0 0) 0x7f96f4001110 con 0x7f970009ac40 2024-03-29T11:24:23.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:24:23.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 1 -- 172.21.15.194:0/2362815230 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f970009da30 con 0x7f970009ac40 2024-03-29T11:24:23.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 1 -- 172.21.15.194:0/2362815230 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483573573 0 0) 0x7f97000b39c0 con 0x7f970009c870 2024-03-29T11:24:23.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.873 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:24:23.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 1 -- 172.21.15.194:0/2362815230 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96f4001110 con 0x7f970009c870 2024-03-29T11:24:23.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 1 -- 172.21.15.194:0/2362815230 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1084640353 0 0) 0x7f96ec003520 con 0x7f97000b4360 2024-03-29T11:24:23.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 1 -- 172.21.15.194:0/2362815230 >> v1:172.21.15.135:6790/0 conn(0x7f970009c870 legacy=0x7f970013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 1 -- 172.21.15.194:0/2362815230 >> v1:172.21.15.135:6789/0 conn(0x7f970009ac40 legacy=0x7f97000b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 10 monclient: _finish_hunting 0 2024-03-29T11:24:23.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 1 monclient: found mon.noname-c 2024-03-29T11:24:23.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:23.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:24:23.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 1 -- 172.21.15.194:0/2362815230 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97000b5160 con 0x7f97000b4360 2024-03-29T11:24:23.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 10 monclient: _finish_auth 0 2024-03-29T11:24:23.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 10 monclient: _check_auth_tickets 2024-03-29T11:24:23.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:23.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 1 -- 172.21.15.194:0/2362815230 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f96f8001a10 con 0x7f97000b4360 2024-03-29T11:24:23.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f97087af640 5 monclient: authenticate success, global_id 4147 2024-03-29T11:24:23.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f97087af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:23.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:23.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 10 monclient: dump: 2024-03-29T11:24:23.875 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.875 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.875 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.875 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.875 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.875 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.875 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.876 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.876 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.876 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f97087af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:23.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 1 -- 172.21.15.194:0/2362815230 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96f8001bf0 con 0x7f97000b4360 2024-03-29T11:24:23.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:23.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f97087af640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:23.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f97087af640 4 set_mon_vals no callback set 2024-03-29T11:24:23.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 1 -- 172.21.15.194:0/2362815230 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f96f8002000 con 0x7f97000b4360 2024-03-29T11:24:23.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:23.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.881+0000 7f9705549640 10 monclient: dump: 2024-03-29T11:24:23.877 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.877 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.877 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.877 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.877 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.877 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.877 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.877 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.877 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.877 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.882+0000 7f97087af640 10 monclient: shutdown 2024-03-29T11:24:23.878 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.882+0000 7f97087af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:23.878 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.882+0000 7f97087af640 1 -- 172.21.15.194:0/2362815230 >> v1:172.21.15.194:6789/0 conn(0x7f97000b4360 legacy=0x7f970014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.878 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.882+0000 7f97087af640 1 -- 172.21.15.194:0/2362815230 shutdown_connections 2024-03-29T11:24:23.878 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.882+0000 7f97087af640 1 -- 172.21.15.194:0/2362815230 >> 172.21.15.194:0/2362815230 conn(0x7f9700066b00 msgr2=0x7f9700066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:23.878 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.882+0000 7f97087af640 1 -- 172.21.15.194:0/2362815230 shutdown_connections 2024-03-29T11:24:23.878 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.882+0000 7f97087af640 1 -- 172.21.15.194:0/2362815230 wait complete. 2024-03-29T11:24:23.878 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.882+0000 7f97087af640 10 monclient: build_initial_monmap 2024-03-29T11:24:23.878 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.882+0000 7f97087af640 10 monclient: monmap: 2024-03-29T11:24:23.878 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-29T11:24:23.879 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:23.879 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 0.000000 2024-03-29T11:24:23.879 INFO:teuthology.orchestra.run.smithi194.stderr:created 0.000000 2024-03-29T11:24:23.879 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:23.879 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.879 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:23.879 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:24:23.879 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:24:23.879 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.879 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.882+0000 7f97087af640 1 librados: starting msgr at 2024-03-29T11:24:23.879 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.882+0000 7f97087af640 1 librados: starting objecter 2024-03-29T11:24:23.879 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f97087af640 1 Processor -- start 2024-03-29T11:24:23.879 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f97087af640 1 -- start start 2024-03-29T11:24:23.879 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f97087af640 1 librados: setting wanted keys 2024-03-29T11:24:23.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f97087af640 1 librados: calling monclient init 2024-03-29T11:24:23.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f97087af640 10 monclient: init 2024-03-29T11:24:23.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f97087af640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:23.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f97087af640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:23.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f97087af640 10 monclient(hunting): picked mon.noname-c con 0x7f970009ac40 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:23.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f97087af640 10 monclient(hunting): picked mon.noname-b con 0x7f9700142c20 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:23.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f97087af640 10 monclient(hunting): picked mon.noname-a con 0x7f97001d7dc0 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:23.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f97087af640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97001d4d30 con 0x7f97001d7dc0 2024-03-29T11:24:23.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f97087af640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97001430b0 con 0x7f9700142c20 2024-03-29T11:24:23.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f97087af640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97001d71b0 con 0x7f970009ac40 2024-03-29T11:24:23.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f97087af640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:23.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f97087af640 10 monclient(hunting): authenticate will time out at 771.169754s 2024-03-29T11:24:23.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f9705d4a640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f9700142c20 0x7f97001d76b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:57380/0 (socket says 172.21.15.194:57380) 2024-03-29T11:24:23.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f9705d4a640 1 -- 172.21.15.194:0/3426156234 learned_addr learned my addr 172.21.15.194:0/3426156234 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-29T11:24:23.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f96f2ffd640 10 client.?.objecter ms_handle_connect 0x7f9700142c20 2024-03-29T11:24:23.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f96f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:23.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f96f2ffd640 10 client.?.objecter ms_handle_connect 0x7f970009ac40 2024-03-29T11:24:23.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f96f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:23.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f96f2ffd640 10 client.?.objecter ms_handle_connect 0x7f97001d7dc0 2024-03-29T11:24:23.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f96f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:23.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f96f2ffd640 1 -- 172.21.15.194:0/3426156234 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4294227696 0 0) 0x7f97001430b0 con 0x7f9700142c20 2024-03-29T11:24:23.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f96f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f96f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f96f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f96f2ffd640 10 monclient(hunting): my global_id is 4150 2024-03-29T11:24:23.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.883+0000 7f96f2ffd640 1 -- 172.21.15.194:0/3426156234 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96e4004eb0 con 0x7f9700142c20 2024-03-29T11:24:23.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 1 -- 172.21.15.194:0/3426156234 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690564267 0 0) 0x7f97001d4d30 con 0x7f97001d7dc0 2024-03-29T11:24:23.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:24:23.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 1 -- 172.21.15.194:0/3426156234 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97001430b0 con 0x7f97001d7dc0 2024-03-29T11:24:23.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 1 -- 172.21.15.194:0/3426156234 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912126128 0 0) 0x7f97001d71b0 con 0x7f970009ac40 2024-03-29T11:24:23.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:23.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:23.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:23.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:24:23.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 1 -- 172.21.15.194:0/3426156234 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97001d4d30 con 0x7f970009ac40 2024-03-29T11:24:23.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 1 -- 172.21.15.194:0/3426156234 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3694471081 0 0) 0x7f96e4004eb0 con 0x7f9700142c20 2024-03-29T11:24:23.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 1 -- 172.21.15.194:0/3426156234 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97001d71b0 con 0x7f9700142c20 2024-03-29T11:24:23.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 1 -- 172.21.15.194:0/3426156234 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f96fc001a10 con 0x7f9700142c20 2024-03-29T11:24:23.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:23.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 monclient(hunting): dump: 2024-03-29T11:24:23.883 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.883 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.883 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.884 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.884 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.884 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.884 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.884 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.884 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.884 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:23.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 1 -- 172.21.15.194:0/3426156234 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1530742299 0 0) 0x7f97001430b0 con 0x7f97001d7dc0 2024-03-29T11:24:23.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 1 -- 172.21.15.194:0/3426156234 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96e4004eb0 con 0x7f97001d7dc0 2024-03-29T11:24:23.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 1 -- 172.21.15.194:0/3426156234 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f96f8001f00 con 0x7f97001d7dc0 2024-03-29T11:24:23.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.885 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:23.885 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 monclient(hunting): dump: 2024-03-29T11:24:23.885 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.885 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.885 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.885 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.885 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.885 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.885 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.885 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.885 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.885 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.886 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 1 -- 172.21.15.194:0/3426156234 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1559846490 0 0) 0x7f97001d4d30 con 0x7f970009ac40 2024-03-29T11:24:23.886 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 1 -- 172.21.15.194:0/3426156234 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97001430b0 con 0x7f970009ac40 2024-03-29T11:24:23.886 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 1 -- 172.21.15.194:0/3426156234 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f96f40018a0 con 0x7f970009ac40 2024-03-29T11:24:23.886 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.886 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:23.886 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 monclient(hunting): dump: 2024-03-29T11:24:23.886 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.886 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.886 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.886 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.887 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.887 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.887 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.887 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.887 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.887 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.887 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 1 -- 172.21.15.194:0/3426156234 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 436131668 0 0) 0x7f97001d71b0 con 0x7f9700142c20 2024-03-29T11:24:23.887 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 1 -- 172.21.15.194:0/3426156234 >> v1:172.21.15.135:6790/0 conn(0x7f970009ac40 legacy=0x7f9700143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.887 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 1 -- 172.21.15.194:0/3426156234 >> v1:172.21.15.135:6789/0 conn(0x7f97001d7dc0 legacy=0x7f9700145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.887 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:23.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 1 monclient: found mon.b 2024-03-29T11:24:23.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:23.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:23.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 1 -- 172.21.15.194:0/3426156234 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9700144ce0 con 0x7f9700142c20 2024-03-29T11:24:23.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:23.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:23.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:23.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f97087af640 5 monclient: authenticate success, global_id 4150 2024-03-29T11:24:23.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f97087af640 10 monclient: _renew_subs 2024-03-29T11:24:23.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f97087af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:23.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f97087af640 1 -- 172.21.15.194:0/3426156234 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9700145490 con 0x7f9700142c20 2024-03-29T11:24:23.889 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f97087af640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:23.889 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f97087af640 10 monclient: _renew_subs 2024-03-29T11:24:23.889 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f97087af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:23.889 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f97087af640 1 -- 172.21.15.194:0/3426156234 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9700146330 con 0x7f9700142c20 2024-03-29T11:24:23.889 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f97087af640 1 librados: init done 2024-03-29T11:24:23.889 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 1 -- 172.21.15.194:0/3426156234 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96fc001c70 con 0x7f9700142c20 2024-03-29T11:24:23.889 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:23.889 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 1 -- 172.21.15.194:0/3426156234 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f96fc003a60 con 0x7f9700142c20 2024-03-29T11:24:23.889 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:23.889 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:23.889 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 monclient: dump: 2024-03-29T11:24:23.889 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:23.889 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:23.889 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.889 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:23.890 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:23.890 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:23.890 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:23.890 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:23.890 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:23.890 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:23.890 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f9704d48640 4 set_mon_vals no callback set 2024-03-29T11:24:23.890 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 1 -- 172.21.15.194:0/3426156234 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f96fc004010 con 0x7f9700142c20 2024-03-29T11:24:23.890 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 1 -- 172.21.15.194:0/3426156234 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 634784349 0 0) 0x7f96fc006990 con 0x7f9700142c20 2024-03-29T11:24:23.890 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 10 client.4150.objecter ms_dispatch 0x7f9700000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:24:23.890 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 3 client.4150.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:24:23.890 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 3 client.4150.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:24:23.890 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.884+0000 7f96f2ffd640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-29T11:24:23.890 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.885+0000 7f97087af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.891 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.885+0000 7f97087af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.891 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.885+0000 7f97087af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:23.891 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.885+0000 7f97087af640 1 -- 172.21.15.194:0/3426156234 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96e8005080 con 0x7f9700142c20 2024-03-29T11:24:23.891 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.887+0000 7f96f2ffd640 1 -- 172.21.15.194:0/3426156234 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f96fc004450 con 0x7f9700142c20 2024-03-29T11:24:23.891 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.887+0000 7f96f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:23.891 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.887+0000 7f96f2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:23.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.959+0000 7f97087af640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b27f71ce-28e4-4daf-b25f-6fe58abcb572", "id": 6}] 2024-03-29T11:24:23.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.959+0000 7f97087af640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b27f71ce-28e4-4daf-b25f-6fe58abcb572", "id": 6}] 2024-03-29T11:24:23.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.959+0000 7f97087af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:23.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.959+0000 7f97087af640 1 -- 172.21.15.194:0/3426156234 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b27f71ce-28e4-4daf-b25f-6fe58abcb572", "id": 6} v 0) v1 -- 0x7f96e80058b0 con 0x7f9700142c20 2024-03-29T11:24:23.945 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.963+0000 7f96f2ffd640 1 -- 172.21.15.194:0/3426156234 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b27f71ce-28e4-4daf-b25f-6fe58abcb572", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 1700065970 0 1113811849) 0x7f96fc0063a0 con 0x7f9700142c20 2024-03-29T11:24:23.945 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.963+0000 7f96f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b27f71ce-28e4-4daf-b25f-6fe58abcb572", "id": 6}] 2024-03-29T11:24:23.945 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.963+0000 7f96f2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:23.945 INFO:teuthology.orchestra.run.smithi194.stdout:6 2024-03-29T11:24:23.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.964+0000 7f97087af640 10 librados: watch_flush enter 2024-03-29T11:24:23.947 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.964+0000 7f97087af640 10 librados: watch_flush exit 2024-03-29T11:24:23.947 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.964+0000 7f97087af640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-29T11:24:23.947 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.964+0000 7f97087af640 10 client.4150.objecter successfully canceled tick 2024-03-29T11:24:23.947 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.964+0000 7f97087af640 10 monclient: shutdown 2024-03-29T11:24:23.947 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.964+0000 7f97087af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:23.947 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.964+0000 7f97087af640 1 -- 172.21.15.194:0/3426156234 >> v1:172.21.15.194:6789/0 conn(0x7f9700142c20 legacy=0x7f97001d76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:23.947 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.964+0000 7f97087af640 1 -- 172.21.15.194:0/3426156234 shutdown_connections 2024-03-29T11:24:23.947 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.964+0000 7f97087af640 1 -- 172.21.15.194:0/3426156234 >> 172.21.15.194:0/3426156234 conn(0x7f9700066b00 msgr2=0x7f970013a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:23.947 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.964+0000 7f97087af640 1 -- 172.21.15.194:0/3426156234 shutdown_connections 2024-03-29T11:24:23.947 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.964+0000 7f97087af640 1 -- 172.21.15.194:0/3426156234 wait complete. 2024-03-29T11:24:23.947 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:23.964+0000 7f97087af640 1 librados: shutdown 2024-03-29T11:24:23.956 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new e19067d4-da68-4171-8fb9-62f960916334 7 2024-03-29T11:24:24.016 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:24.034+0000 7f992383e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:24:24.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.075+0000 7fd356d8f640 10 monclient: get_monmap_and_config 2024-03-29T11:24:24.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.075+0000 7fd356d8f640 10 monclient: build_initial_monmap 2024-03-29T11:24:24.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.075+0000 7fd356d8f640 10 monclient: monmap: 2024-03-29T11:24:24.057 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-29T11:24:24.057 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:24.057 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:24:24.076482+0000 2024-03-29T11:24:24.058 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:24:24.076482+0000 2024-03-29T11:24:24.058 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:24.058 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:24.058 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:24.058 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:24:24.058 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:24:24.058 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:24.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.075+0000 7fd356d8f640 1 Processor -- start 2024-03-29T11:24:24.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.075+0000 7fd356d8f640 1 -- start start 2024-03-29T11:24:24.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.075+0000 7fd356d8f640 10 monclient: init 2024-03-29T11:24:24.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd356d8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:24.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd356d8f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:24.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd356d8f640 10 monclient(hunting): picked mon.noname-a con 0x7fd35009ac40 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:24.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd356d8f640 10 monclient(hunting): picked mon.noname-b con 0x7fd35009c870 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:24.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd356d8f640 10 monclient(hunting): picked mon.noname-c con 0x7fd3500b4360 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:24.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd356d8f640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35009dec0 con 0x7fd35009ac40 2024-03-29T11:24:24.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd356d8f640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35009da30 con 0x7fd3500b4360 2024-03-29T11:24:24.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd356d8f640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3500b39c0 con 0x7fd35009c870 2024-03-29T11:24:24.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd356d8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:24.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd356d8f640 10 monclient(hunting): authenticate will time out at 771.362552s 2024-03-29T11:24:24.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd35532c640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fd3500b4360 0x7fd35014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:57394/0 (socket says 172.21.15.194:57394) 2024-03-29T11:24:24.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd35532c640 1 -- 172.21.15.194:0/3146203748 learned_addr learned my addr 172.21.15.194:0/3146203748 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-29T11:24:24.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd34f7fe640 1 -- 172.21.15.194:0/3146203748 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842668482 0 0) 0x7fd35009da30 con 0x7fd3500b4360 2024-03-29T11:24:24.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd34f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:24.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd34f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:24.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd34f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:24.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd34f7fe640 10 monclient(hunting): my global_id is 4153 2024-03-29T11:24:24.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd34f7fe640 1 -- 172.21.15.194:0/3146203748 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd340003520 con 0x7fd3500b4360 2024-03-29T11:24:24.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd34f7fe640 1 -- 172.21.15.194:0/3146203748 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121519178 0 0) 0x7fd344001110 con 0x7fd35009ac40 2024-03-29T11:24:24.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd34f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:24.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd34f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:24.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd34f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:24.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd34f7fe640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:24:24.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd34f7fe640 1 -- 172.21.15.194:0/3146203748 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd35009da30 con 0x7fd35009ac40 2024-03-29T11:24:24.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd34f7fe640 1 -- 172.21.15.194:0/3146203748 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212193539 0 0) 0x7fd3500b39c0 con 0x7fd35009c870 2024-03-29T11:24:24.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd34f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:24.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd34f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:24.069 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd34f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:24.069 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd34f7fe640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:24:24.069 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.076+0000 7fd34f7fe640 1 -- 172.21.15.194:0/3146203748 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd344001110 con 0x7fd35009c870 2024-03-29T11:24:24.069 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd34f7fe640 1 -- 172.21.15.194:0/3146203748 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2316552846 0 0) 0x7fd340003520 con 0x7fd3500b4360 2024-03-29T11:24:24.069 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd34f7fe640 1 -- 172.21.15.194:0/3146203748 >> v1:172.21.15.135:6790/0 conn(0x7fd35009c870 legacy=0x7fd35013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.069 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd34f7fe640 1 -- 172.21.15.194:0/3146203748 >> v1:172.21.15.135:6789/0 conn(0x7fd35009ac40 legacy=0x7fd3500b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.069 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd34f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:24.069 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd34f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:24:24.069 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd34f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:24.069 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd34f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:24:24.069 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd34f7fe640 1 -- 172.21.15.194:0/3146203748 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3500b5160 con 0x7fd3500b4360 2024-03-29T11:24:24.069 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd34f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:24.069 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd34f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:24.069 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd34f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:24.070 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd356d8f640 5 monclient: authenticate success, global_id 4153 2024-03-29T11:24:24.070 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd356d8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:24.070 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd34f7fe640 1 -- 172.21.15.194:0/3146203748 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fd348001a10 con 0x7fd3500b4360 2024-03-29T11:24:24.070 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd34f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.070 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd34f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:24.070 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd34f7fe640 10 monclient: dump: 2024-03-29T11:24:24.070 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:24.070 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:24.070 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:24.070 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:24.070 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.070 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:24.070 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:24.071 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:24.071 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:24.071 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:24.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd356d8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:24.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd34f7fe640 1 -- 172.21.15.194:0/3146203748 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd348001c60 con 0x7fd3500b4360 2024-03-29T11:24:24.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd34f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:24.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd356d8f640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:24.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd34f7fe640 1 -- 172.21.15.194:0/3146203748 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fd348001ff0 con 0x7fd3500b4360 2024-03-29T11:24:24.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd34f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd356d8f640 4 set_mon_vals no callback set 2024-03-29T11:24:24.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd34f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:24.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd34f7fe640 10 monclient: dump: 2024-03-29T11:24:24.071 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:24.071 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:24.071 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:24.072 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:24.072 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.072 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:24.072 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:24.072 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:24.072 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:24.072 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:24.072 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd356d8f640 10 monclient: shutdown 2024-03-29T11:24:24.072 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd356d8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:24.072 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd356d8f640 1 -- 172.21.15.194:0/3146203748 >> v1:172.21.15.194:6789/0 conn(0x7fd3500b4360 legacy=0x7fd35014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.072 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd356d8f640 1 -- 172.21.15.194:0/3146203748 shutdown_connections 2024-03-29T11:24:24.072 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd356d8f640 1 -- 172.21.15.194:0/3146203748 >> 172.21.15.194:0/3146203748 conn(0x7fd350066b00 msgr2=0x7fd350066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:24.072 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd356d8f640 1 -- 172.21.15.194:0/3146203748 shutdown_connections 2024-03-29T11:24:24.072 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.077+0000 7fd356d8f640 1 -- 172.21.15.194:0/3146203748 wait complete. 2024-03-29T11:24:24.073 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.078+0000 7fd356d8f640 10 monclient: build_initial_monmap 2024-03-29T11:24:24.073 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.078+0000 7fd356d8f640 10 monclient: monmap: 2024-03-29T11:24:24.073 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-03-29T11:24:24.073 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:24.073 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 0.000000 2024-03-29T11:24:24.073 INFO:teuthology.orchestra.run.smithi194.stderr:created 0.000000 2024-03-29T11:24:24.073 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:24.073 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:24.073 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:24.073 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:24:24.073 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:24:24.073 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:24.073 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.078+0000 7fd356d8f640 1 librados: starting msgr at 2024-03-29T11:24:24.073 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.078+0000 7fd356d8f640 1 librados: starting objecter 2024-03-29T11:24:24.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.078+0000 7fd356d8f640 1 Processor -- start 2024-03-29T11:24:24.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.078+0000 7fd356d8f640 1 -- start start 2024-03-29T11:24:24.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.078+0000 7fd356d8f640 1 librados: setting wanted keys 2024-03-29T11:24:24.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.078+0000 7fd356d8f640 1 librados: calling monclient init 2024-03-29T11:24:24.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.078+0000 7fd356d8f640 10 monclient: init 2024-03-29T11:24:24.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd356d8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:24.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd356d8f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:24.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd356d8f640 10 monclient(hunting): picked mon.noname-c con 0x7fd35009ac40 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:24.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd356d8f640 10 monclient(hunting): picked mon.noname-a con 0x7fd350145290 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:24.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd356d8f640 10 monclient(hunting): picked mon.noname-b con 0x7fd3501d7880 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:24.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd356d8f640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3501457b0 con 0x7fd350145290 2024-03-29T11:24:24.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd356d8f640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd350145c40 con 0x7fd3501d7880 2024-03-29T11:24:24.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd356d8f640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd350145dd0 con 0x7fd35009ac40 2024-03-29T11:24:24.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd356d8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:24.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd356d8f640 10 monclient(hunting): authenticate will time out at 771.365653s 2024-03-29T11:24:24.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd35532c640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fd3501d7880 0x7fd3500b14c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:57404/0 (socket says 172.21.15.194:57404) 2024-03-29T11:24:24.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd35532c640 1 -- 172.21.15.194:0/831296478 learned_addr learned my addr 172.21.15.194:0/831296478 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-03-29T11:24:24.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd34d7fa640 10 client.?.objecter ms_handle_connect 0x7fd3501d7880 2024-03-29T11:24:24.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd34d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:24.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd34d7fa640 10 client.?.objecter ms_handle_connect 0x7fd35009ac40 2024-03-29T11:24:24.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd34d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:24.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd34d7fa640 10 client.?.objecter ms_handle_connect 0x7fd350145290 2024-03-29T11:24:24.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd34d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:24.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd34d7fa640 1 -- 172.21.15.194:0/831296478 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40191150 0 0) 0x7fd350145c40 con 0x7fd3501d7880 2024-03-29T11:24:24.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd34d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:24.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd34d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:24.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd34d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:24.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd34d7fa640 10 monclient(hunting): my global_id is 4156 2024-03-29T11:24:24.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd34d7fa640 1 -- 172.21.15.194:0/831296478 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd334004eb0 con 0x7fd3501d7880 2024-03-29T11:24:24.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd34d7fa640 1 -- 172.21.15.194:0/831296478 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839751099 0 0) 0x7fd3501457b0 con 0x7fd350145290 2024-03-29T11:24:24.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd34d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:24.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd34d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:24.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd34d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:24.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd34d7fa640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:24:24.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd34d7fa640 1 -- 172.21.15.194:0/831296478 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd350145c40 con 0x7fd350145290 2024-03-29T11:24:24.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd34d7fa640 1 -- 172.21.15.194:0/831296478 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3825856876 0 0) 0x7fd350145dd0 con 0x7fd35009ac40 2024-03-29T11:24:24.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd34d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:24.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd34d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:24.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd34d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:24.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd34d7fa640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:24:24.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.079+0000 7fd34d7fa640 1 -- 172.21.15.194:0/831296478 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3501457b0 con 0x7fd35009ac40 2024-03-29T11:24:24.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 1 -- 172.21.15.194:0/831296478 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1663677185 0 0) 0x7fd334004eb0 con 0x7fd3501d7880 2024-03-29T11:24:24.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 1 -- 172.21.15.194:0/831296478 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd350145dd0 con 0x7fd3501d7880 2024-03-29T11:24:24.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 1 -- 172.21.15.194:0/831296478 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fd348001f70 con 0x7fd3501d7880 2024-03-29T11:24:24.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:24.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 10 monclient(hunting): dump: 2024-03-29T11:24:24.077 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:24.077 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:24.077 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:24.077 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:24.077 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.077 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:24.077 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:24.077 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:24.077 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:24.077 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:24.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:24.078 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 1 -- 172.21.15.194:0/831296478 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3798027180 0 0) 0x7fd350145c40 con 0x7fd350145290 2024-03-29T11:24:24.078 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 1 -- 172.21.15.194:0/831296478 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd334004eb0 con 0x7fd350145290 2024-03-29T11:24:24.078 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 1 -- 172.21.15.194:0/831296478 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fd33c001820 con 0x7fd350145290 2024-03-29T11:24:24.078 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.078 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:24.078 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 10 monclient(hunting): dump: 2024-03-29T11:24:24.078 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:24.078 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:24.078 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:24.078 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:24.078 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.078 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:24.078 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:24.078 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:24.078 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:24.079 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:24.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 1 -- 172.21.15.194:0/831296478 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3825267085 0 0) 0x7fd3501457b0 con 0x7fd35009ac40 2024-03-29T11:24:24.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 1 -- 172.21.15.194:0/831296478 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd350145c40 con 0x7fd35009ac40 2024-03-29T11:24:24.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 1 -- 172.21.15.194:0/831296478 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fd3440018a0 con 0x7fd35009ac40 2024-03-29T11:24:24.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:24.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 10 monclient(hunting): dump: 2024-03-29T11:24:24.079 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:24.079 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:24.079 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:24.079 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:24.079 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.079 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:24.079 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:24.080 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:24.080 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:24.080 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:24.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 1 -- 172.21.15.194:0/831296478 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3155185740 0 0) 0x7fd350145dd0 con 0x7fd3501d7880 2024-03-29T11:24:24.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 1 -- 172.21.15.194:0/831296478 >> v1:172.21.15.135:6790/0 conn(0x7fd35009ac40 legacy=0x7fd3501d7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 1 -- 172.21.15.194:0/831296478 >> v1:172.21.15.135:6789/0 conn(0x7fd350145290 legacy=0x7fd3500b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:24:24.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 1 monclient: found mon.b 2024-03-29T11:24:24.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:24.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:24.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 1 -- 172.21.15.194:0/831296478 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd350145f60 con 0x7fd3501d7880 2024-03-29T11:24:24.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:24:24.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:24:24.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:24.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd356d8f640 5 monclient: authenticate success, global_id 4156 2024-03-29T11:24:24.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd356d8f640 10 monclient: _renew_subs 2024-03-29T11:24:24.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd356d8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:24.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd356d8f640 1 -- 172.21.15.194:0/831296478 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd350144d40 con 0x7fd3501d7880 2024-03-29T11:24:24.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd356d8f640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:24.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd356d8f640 10 monclient: _renew_subs 2024-03-29T11:24:24.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd356d8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:24.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd356d8f640 1 -- 172.21.15.194:0/831296478 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3500b76b0 con 0x7fd3501d7880 2024-03-29T11:24:24.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd356d8f640 1 librados: init done 2024-03-29T11:24:24.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 1 -- 172.21.15.194:0/831296478 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd348004e10 con 0x7fd3501d7880 2024-03-29T11:24:24.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:24.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 1 -- 172.21.15.194:0/831296478 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fd348001530 con 0x7fd3501d7880 2024-03-29T11:24:24.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:24.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 10 monclient: dump: 2024-03-29T11:24:24.082 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-03-29T11:24:24.082 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:24.082 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:24.082 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:24.082 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.082 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-03-29T11:24:24.082 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:24.082 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:24.082 INFO:teuthology.orchestra.run.smithi194.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:24.082 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:24:24.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34effd640 4 set_mon_vals no callback set 2024-03-29T11:24:24.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 1 -- 172.21.15.194:0/831296478 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd3480054f0 con 0x7fd3501d7880 2024-03-29T11:24:24.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 1 -- 172.21.15.194:0/831296478 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 3037573528 0 0) 0x7fd348006d80 con 0x7fd3501d7880 2024-03-29T11:24:24.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 10 client.4156.objecter ms_dispatch 0x7fd350000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:24:24.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 3 client.4156.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:24:24.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 3 client.4156.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:24:24.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.080+0000 7fd34d7fa640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-29T11:24:24.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.081+0000 7fd356d8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:24.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.081+0000 7fd356d8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:24.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.081+0000 7fd356d8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:24.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.081+0000 7fd356d8f640 1 -- 172.21.15.194:0/831296478 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd330005080 con 0x7fd3501d7880 2024-03-29T11:24:24.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.083+0000 7fd34d7fa640 1 -- 172.21.15.194:0/831296478 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd3480067d0 con 0x7fd3501d7880 2024-03-29T11:24:24.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.083+0000 7fd34d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:24.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.083+0000 7fd34d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:24.083 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-29T11:24:24.083+0000 7fe5b562c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:24:24.135 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.154+0000 7fd356d8f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e19067d4-da68-4171-8fb9-62f960916334", "id": 7}] 2024-03-29T11:24:24.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.154+0000 7fd356d8f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e19067d4-da68-4171-8fb9-62f960916334", "id": 7}] 2024-03-29T11:24:24.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.154+0000 7fd356d8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:24.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.154+0000 7fd356d8f640 1 -- 172.21.15.194:0/831296478 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd new", "uuid": "e19067d4-da68-4171-8fb9-62f960916334", "id": 7} v 0) v1 -- 0x7fd3300058b0 con 0x7fd3501d7880 2024-03-29T11:24:24.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.157+0000 7fd34d7fa640 1 -- 172.21.15.194:0/831296478 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e19067d4-da68-4171-8fb9-62f960916334", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 1395955473 0 1371665406) 0x7fd348006ac0 con 0x7fd3501d7880 2024-03-29T11:24:24.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.157+0000 7fd34d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e19067d4-da68-4171-8fb9-62f960916334", "id": 7}] 2024-03-29T11:24:24.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.157+0000 7fd34d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:24.139 INFO:teuthology.orchestra.run.smithi194.stdout:7 2024-03-29T11:24:24.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.158+0000 7fd356d8f640 10 librados: watch_flush enter 2024-03-29T11:24:24.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.158+0000 7fd356d8f640 10 librados: watch_flush exit 2024-03-29T11:24:24.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.158+0000 7fd356d8f640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-29T11:24:24.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.158+0000 7fd356d8f640 10 client.4156.objecter successfully canceled tick 2024-03-29T11:24:24.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.158+0000 7fd356d8f640 10 monclient: shutdown 2024-03-29T11:24:24.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.158+0000 7fd356d8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:24.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.158+0000 7fd356d8f640 1 -- 172.21.15.194:0/831296478 >> v1:172.21.15.194:6789/0 conn(0x7fd3501d7880 legacy=0x7fd3500b14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.158+0000 7fd356d8f640 1 -- 172.21.15.194:0/831296478 shutdown_connections 2024-03-29T11:24:24.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.158+0000 7fd356d8f640 1 -- 172.21.15.194:0/831296478 >> 172.21.15.194:0/831296478 conn(0x7fd350066b00 msgr2=0x7fd350139300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:24.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.158+0000 7fd356d8f640 1 -- 172.21.15.194:0/831296478 shutdown_connections 2024-03-29T11:24:24.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.158+0000 7fd356d8f640 1 -- 172.21.15.194:0/831296478 wait complete. 2024-03-29T11:24:24.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-29T11:24:24.158+0000 7fd356d8f640 1 librados: shutdown 2024-03-29T11:24:24.150 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:24:24.151 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:24:24.152 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:24.170+0000 7f992383e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:24:24.153 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:24:24.153 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:24:24.153 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:24:24.155 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:24:24.155 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:24:24.155 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:24:24.157 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:24:24.157 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:24:24.157 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:24:24.159 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:24:24.159 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:24:24.159 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:24:24.161 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:24:24.161 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:24:24.161 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:24:24.163 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:24:24.163 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:24:24.163 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:24:24.165 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:24:24.165 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:24:24.166 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:24:24.167 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:24:24.168 DEBUG:teuthology.orchestra.run.smithi135:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:24.226 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-29T11:24:24.244+0000 7fe5b562c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:24:24.292 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.310+0000 7f52b6c82640 10 monclient: get_monmap_and_config 2024-03-29T11:24:24.292 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.310+0000 7f52b6c82640 10 monclient: build_initial_monmap 2024-03-29T11:24:24.292 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.310+0000 7f52b6c82640 10 monclient: monmap: 2024-03-29T11:24:24.292 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2024-03-29T11:24:24.292 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:24.292 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-03-29T11:24:24.311843+0000 2024-03-29T11:24:24.292 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-03-29T11:24:24.311843+0000 2024-03-29T11:24:24.292 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:24.293 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:24.293 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:24.293 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:24:24.293 INFO:teuthology.orchestra.run.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:24:24.293 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-03-29T11:24:24.299 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.311+0000 7f52b6c82640 1 Processor -- start 2024-03-29T11:24:24.300 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.311+0000 7f52b6c82640 1 -- start start 2024-03-29T11:24:24.300 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.311+0000 7f52b6c82640 10 monclient: init 2024-03-29T11:24:24.300 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.311+0000 7f52b6c82640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:24.300 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.311+0000 7f52b6c82640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:24.300 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.311+0000 7f52b6c82640 10 monclient(hunting): picked mon.noname-a con 0x7f52b009ac40 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:24.300 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.311+0000 7f52b6c82640 10 monclient(hunting): picked mon.noname-c con 0x7f52b009c870 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:24.300 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.311+0000 7f52b6c82640 10 monclient(hunting): picked mon.noname-b con 0x7f52b00b4360 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:24.300 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.311+0000 7f52b6c82640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b009dec0 con 0x7f52b009ac40 2024-03-29T11:24:24.300 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.311+0000 7f52b6c82640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b009da30 con 0x7f52b009c870 2024-03-29T11:24:24.300 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.311+0000 7f52b6c82640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b00b39c0 con 0x7f52b00b4360 2024-03-29T11:24:24.300 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.311+0000 7f52b6c82640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:24.300 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.311+0000 7f52b6c82640 10 monclient(hunting): authenticate will time out at 771.680749s 2024-03-29T11:24:24.300 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.311+0000 7f52b4a1e640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f52b009ac40 0x7f52b00b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:40048/0 (socket says 172.21.15.135:40048) 2024-03-29T11:24:24.301 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.311+0000 7f52b521f640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f52b00b4360 0x7f52b014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:35146/0 (socket says 172.21.15.135:35146) 2024-03-29T11:24:24.301 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.311+0000 7f52b4a1e640 1 -- 172.21.15.135:0/2731666936 learned_addr learned my addr 172.21.15.135:0/2731666936 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:24:24.301 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 1 -- 172.21.15.135:0/2731666936 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1148932902 0 0) 0x7f52b00b39c0 con 0x7f52b00b4360 2024-03-29T11:24:24.301 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:24.301 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:24.301 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:24.301 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 10 monclient(hunting): my global_id is 4178 2024-03-29T11:24:24.301 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 1 -- 172.21.15.135:0/2731666936 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52a0003520 con 0x7f52b00b4360 2024-03-29T11:24:24.301 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 1 -- 172.21.15.135:0/2731666936 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997493828 0 0) 0x7f52a4001110 con 0x7f52b009ac40 2024-03-29T11:24:24.301 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:24.301 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:24.301 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:24.301 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 10 monclient(hunting): my global_id is 4179 2024-03-29T11:24:24.301 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 1 -- 172.21.15.135:0/2731666936 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52b00b39c0 con 0x7f52b009ac40 2024-03-29T11:24:24.301 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 1 -- 172.21.15.135:0/2731666936 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1079684507 0 0) 0x7f52b009da30 con 0x7f52b009c870 2024-03-29T11:24:24.302 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:24.302 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:24.302 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:24.302 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 10 monclient(hunting): my global_id is 4183 2024-03-29T11:24:24.302 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 1 -- 172.21.15.135:0/2731666936 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52a4001110 con 0x7f52b009c870 2024-03-29T11:24:24.302 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 1 -- 172.21.15.135:0/2731666936 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1777811995 0 0) 0x7f52a0003520 con 0x7f52b00b4360 2024-03-29T11:24:24.302 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 1 -- 172.21.15.135:0/2731666936 >> v1:172.21.15.194:6789/0 conn(0x7f52b009c870 legacy=0x7f52b013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.302 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 1 -- 172.21.15.135:0/2731666936 >> v1:172.21.15.135:6789/0 conn(0x7f52b009ac40 legacy=0x7f52b00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.302 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:24.302 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 1 monclient: found mon.noname-b 2024-03-29T11:24:24.302 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:24.302 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:24:24.302 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 1 -- 172.21.15.135:0/2731666936 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b00b5160 con 0x7f52b00b4360 2024-03-29T11:24:24.302 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:24.303 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:24.303 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:24.303 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 1 -- 172.21.15.135:0/2731666936 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f52a8001a10 con 0x7f52b00b4360 2024-03-29T11:24:24.303 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.303 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:24.303 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 10 monclient: dump: 2024-03-29T11:24:24.303 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-03-29T11:24:24.303 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:24.303 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:24.303 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:24.303 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.303 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:24.303 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:24.304 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:24.304 INFO:teuthology.orchestra.run.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:24.304 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-03-29T11:24:24.304 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52b6c82640 5 monclient: authenticate success, global_id 4178 2024-03-29T11:24:24.304 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52b6c82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:24.304 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 1 -- 172.21.15.135:0/2731666936 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52a8001b70 con 0x7f52b00b4360 2024-03-29T11:24:24.304 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:24.304 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52b6c82640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:24.304 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52b6c82640 4 set_mon_vals no callback set 2024-03-29T11:24:24.304 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 1 -- 172.21.15.135:0/2731666936 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f52a8001fc0 con 0x7f52b00b4360 2024-03-29T11:24:24.304 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.304 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:24.304 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52af7fe640 10 monclient: dump: 2024-03-29T11:24:24.304 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-03-29T11:24:24.304 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:24.305 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:24.305 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:24.305 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.305 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:24.305 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:24.305 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:24.305 INFO:teuthology.orchestra.run.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:24.305 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-03-29T11:24:24.305 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52b6c82640 10 monclient: shutdown 2024-03-29T11:24:24.305 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52b6c82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:24.305 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.312+0000 7f52b6c82640 1 -- 172.21.15.135:0/2731666936 >> v1:172.21.15.135:6790/0 conn(0x7f52b00b4360 legacy=0x7f52b014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.305 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.313+0000 7f52b6c82640 1 -- 172.21.15.135:0/2731666936 shutdown_connections 2024-03-29T11:24:24.305 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.313+0000 7f52b6c82640 1 -- 172.21.15.135:0/2731666936 >> 172.21.15.135:0/2731666936 conn(0x7f52b0066b00 msgr2=0x7f52b0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:24.305 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.313+0000 7f52b6c82640 1 -- 172.21.15.135:0/2731666936 shutdown_connections 2024-03-29T11:24:24.305 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.313+0000 7f52b6c82640 1 -- 172.21.15.135:0/2731666936 wait complete. 2024-03-29T11:24:24.306 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.313+0000 7f52b6c82640 10 monclient: build_initial_monmap 2024-03-29T11:24:24.306 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.313+0000 7f52b6c82640 10 monclient: monmap: 2024-03-29T11:24:24.306 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 0 2024-03-29T11:24:24.306 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:24.306 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 0.000000 2024-03-29T11:24:24.306 INFO:teuthology.orchestra.run.smithi135.stderr:created 0.000000 2024-03-29T11:24:24.306 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:24.306 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:24.306 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:24.306 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:24:24.306 INFO:teuthology.orchestra.run.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:24:24.306 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-03-29T11:24:24.306 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.313+0000 7f52b6c82640 1 librados: starting msgr at 2024-03-29T11:24:24.306 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.313+0000 7f52b6c82640 1 librados: starting objecter 2024-03-29T11:24:24.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.314+0000 7f52b6c82640 1 Processor -- start 2024-03-29T11:24:24.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.314+0000 7f52b6c82640 1 -- start start 2024-03-29T11:24:24.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.314+0000 7f52b6c82640 1 librados: setting wanted keys 2024-03-29T11:24:24.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.314+0000 7f52b6c82640 1 librados: calling monclient init 2024-03-29T11:24:24.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.314+0000 7f52b6c82640 10 monclient: init 2024-03-29T11:24:24.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.314+0000 7f52b6c82640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:24.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.314+0000 7f52b6c82640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:24.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.314+0000 7f52b6c82640 10 monclient(hunting): picked mon.noname-a con 0x7f52b009ac40 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:24.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.314+0000 7f52b6c82640 10 monclient(hunting): picked mon.noname-b con 0x7f52b01408b0 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:24.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.314+0000 7f52b6c82640 10 monclient(hunting): picked mon.noname-c con 0x7f52b01394b0 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:24.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.314+0000 7f52b4a1e640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f52b009ac40 0x7f52b0138da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:40062/0 (socket says 172.21.15.135:40062) 2024-03-29T11:24:24.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.314+0000 7f52b4a1e640 1 -- 172.21.15.135:0/1610067259 learned_addr learned my addr 172.21.15.135:0/1610067259 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:24:24.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.314+0000 7f52b6c82640 1 -- 172.21.15.135:0/1610067259 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b0142770 con 0x7f52b009ac40 2024-03-29T11:24:24.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.314+0000 7f52b6c82640 1 -- 172.21.15.135:0/1610067259 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b01429c0 con 0x7f52b01408b0 2024-03-29T11:24:24.307 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.314+0000 7f52b6c82640 1 -- 172.21.15.135:0/1610067259 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b0142ca0 con 0x7f52b01394b0 2024-03-29T11:24:24.308 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.314+0000 7f52b6c82640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:24.308 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.314+0000 7f52ad7fa640 10 client.?.objecter ms_handle_connect 0x7f52b01394b0 2024-03-29T11:24:24.308 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.314+0000 7f52ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:24.308 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.314+0000 7f52ad7fa640 10 client.?.objecter ms_handle_connect 0x7f52b009ac40 2024-03-29T11:24:24.308 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.314+0000 7f52ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:24.308 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.314+0000 7f52b6c82640 10 monclient(hunting): authenticate will time out at 771.684020s 2024-03-29T11:24:24.308 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 client.?.objecter ms_handle_connect 0x7f52b01408b0 2024-03-29T11:24:24.308 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:24.308 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 1 -- 172.21.15.135:0/1610067259 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044395170 0 0) 0x7f52b0142ca0 con 0x7f52b01394b0 2024-03-29T11:24:24.308 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:24.308 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:24.308 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:24.308 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient(hunting): my global_id is 4181 2024-03-29T11:24:24.308 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 1 -- 172.21.15.135:0/1610067259 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5294004eb0 con 0x7f52b01394b0 2024-03-29T11:24:24.308 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 1 -- 172.21.15.135:0/1610067259 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554758986 0 0) 0x7f52b0142770 con 0x7f52b009ac40 2024-03-29T11:24:24.309 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:24.309 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:24.309 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:24.309 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient(hunting): my global_id is 4182 2024-03-29T11:24:24.309 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 1 -- 172.21.15.135:0/1610067259 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52b0142ca0 con 0x7f52b009ac40 2024-03-29T11:24:24.309 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 1 -- 172.21.15.135:0/1610067259 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054853841 0 0) 0x7f52b01429c0 con 0x7f52b01408b0 2024-03-29T11:24:24.309 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:24.309 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:24.309 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:24.309 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient(hunting): my global_id is 4186 2024-03-29T11:24:24.309 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 1 -- 172.21.15.135:0/1610067259 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52b0142770 con 0x7f52b01408b0 2024-03-29T11:24:24.309 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 1 -- 172.21.15.135:0/1610067259 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2906576333 0 0) 0x7f5294004eb0 con 0x7f52b01394b0 2024-03-29T11:24:24.309 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 1 -- 172.21.15.135:0/1610067259 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52b01429c0 con 0x7f52b01394b0 2024-03-29T11:24:24.309 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 1 -- 172.21.15.135:0/1610067259 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f52a8001e80 con 0x7f52b01394b0 2024-03-29T11:24:24.310 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.310 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:24.310 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient(hunting): dump: 2024-03-29T11:24:24.310 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-03-29T11:24:24.310 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:24.310 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:24.310 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:24.310 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.310 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:24.310 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:24.310 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:24.310 INFO:teuthology.orchestra.run.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:24.310 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-03-29T11:24:24.310 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:24.310 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 1 -- 172.21.15.135:0/1610067259 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3336932497 0 0) 0x7f52b0142770 con 0x7f52b01408b0 2024-03-29T11:24:24.311 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 1 -- 172.21.15.135:0/1610067259 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5294004eb0 con 0x7f52b01408b0 2024-03-29T11:24:24.311 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 1 -- 172.21.15.135:0/1610067259 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f529c001870 con 0x7f52b01408b0 2024-03-29T11:24:24.311 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.311 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:24.311 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient(hunting): dump: 2024-03-29T11:24:24.311 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-03-29T11:24:24.311 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:24.311 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:24.311 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:24.311 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.311 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:24.311 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:24.311 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:24.312 INFO:teuthology.orchestra.run.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:24.312 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-03-29T11:24:24.312 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 1 -- 172.21.15.135:0/1610067259 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1024159079 0 0) 0x7f52b0142ca0 con 0x7f52b009ac40 2024-03-29T11:24:24.312 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 1 -- 172.21.15.135:0/1610067259 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52b0142770 con 0x7f52b009ac40 2024-03-29T11:24:24.312 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 1 -- 172.21.15.135:0/1610067259 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f52a4001b00 con 0x7f52b009ac40 2024-03-29T11:24:24.312 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.312 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:24.312 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient(hunting): dump: 2024-03-29T11:24:24.312 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-03-29T11:24:24.312 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:24.312 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:24.312 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:24.312 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.312 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:24.312 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:24.313 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:24.313 INFO:teuthology.orchestra.run.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:24.313 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-03-29T11:24:24.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 1 -- 172.21.15.135:0/1610067259 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1147723342 0 0) 0x7f5294004eb0 con 0x7f52b01408b0 2024-03-29T11:24:24.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 1 -- 172.21.15.135:0/1610067259 >> v1:172.21.15.135:6790/0 conn(0x7f52b01394b0 legacy=0x7f52b01e2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 1 -- 172.21.15.135:0/1610067259 >> v1:172.21.15.135:6789/0 conn(0x7f52b009ac40 legacy=0x7f52b0138da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:24:24.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 1 monclient: found mon.b 2024-03-29T11:24:24.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:24.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:24.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 1 -- 172.21.15.135:0/1610067259 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b0141bd0 con 0x7f52b01408b0 2024-03-29T11:24:24.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient: _finish_auth 0 2024-03-29T11:24:24.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:24:24.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:24.313 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52b6c82640 5 monclient: authenticate success, global_id 4186 2024-03-29T11:24:24.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52b6c82640 10 monclient: _renew_subs 2024-03-29T11:24:24.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52b6c82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:24.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52b6c82640 1 -- 172.21.15.135:0/1610067259 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52b0141510 con 0x7f52b01408b0 2024-03-29T11:24:24.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52b6c82640 10 client.4186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:24.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52b6c82640 10 monclient: _renew_subs 2024-03-29T11:24:24.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52b6c82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:24.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52b6c82640 1 -- 172.21.15.135:0/1610067259 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b01d77b0 con 0x7f52b01408b0 2024-03-29T11:24:24.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52b6c82640 1 librados: init done 2024-03-29T11:24:24.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 1 -- 172.21.15.135:0/1610067259 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f529c001c80 con 0x7f52b01408b0 2024-03-29T11:24:24.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:24.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 1 -- 172.21.15.135:0/1610067259 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f529c0037b0 con 0x7f52b01408b0 2024-03-29T11:24:24.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:24.314 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 monclient: dump: 2024-03-29T11:24:24.315 INFO:teuthology.orchestra.run.smithi135.stderr:epoch 1 2024-03-29T11:24:24.315 INFO:teuthology.orchestra.run.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:24.315 INFO:teuthology.orchestra.run.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:24.315 INFO:teuthology.orchestra.run.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:24.315 INFO:teuthology.orchestra.run.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.315 INFO:teuthology.orchestra.run.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:24.315 INFO:teuthology.orchestra.run.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:24.315 INFO:teuthology.orchestra.run.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:24.315 INFO:teuthology.orchestra.run.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:24.315 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-03-29T11:24:24.315 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52aeffd640 4 set_mon_vals no callback set 2024-03-29T11:24:24.315 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 1 -- 172.21.15.135:0/1610067259 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f529c003ca0 con 0x7f52b01408b0 2024-03-29T11:24:24.315 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 1 -- 172.21.15.135:0/1610067259 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f529c0055c0 con 0x7f52b01408b0 2024-03-29T11:24:24.315 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 10 client.4186.objecter ms_dispatch 0x7f52b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:24.315 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 3 client.4186.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:24.316 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 3 client.4186.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:24.316 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.315+0000 7f52ad7fa640 20 client.4186.objecter dump_active .. 0 homeless 2024-03-29T11:24:24.316 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.316+0000 7f52b6c82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:24.316 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.316+0000 7f52b6c82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:24.316 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.316+0000 7f52b6c82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:24.316 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.316+0000 7f52b6c82640 1 -- 172.21.15.135:0/1610067259 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52b0139900 con 0x7f52b01408b0 2024-03-29T11:24:24.316 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.318+0000 7f52ad7fa640 1 -- 172.21.15.135:0/1610067259 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f529c004ef0 con 0x7f52b01408b0 2024-03-29T11:24:24.316 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.318+0000 7f52ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:24.316 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.318+0000 7f52ad7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:24.317 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:24.322+0000 7f992383e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:24:24.348 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-29T11:24:24.366+0000 7fe5b562c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:24:24.350 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:24.368+0000 7f992383e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:24:24.360 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.378+0000 7f52b6c82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:24.360 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.378+0000 7f52b6c82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:24.360 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.378+0000 7f52b6c82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:24.360 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.378+0000 7f52b6c82640 1 -- 172.21.15.135:0/1610067259 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52b01419f0 con 0x7f52b01408b0 2024-03-29T11:24:24.360 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.378+0000 7f52ad7fa640 1 -- 172.21.15.135:0/1610067259 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f529c0051e0 con 0x7f52b01408b0 2024-03-29T11:24:24.360 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.378+0000 7f52ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:24.360 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.378+0000 7f52ad7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:24.361 INFO:teuthology.orchestra.run.smithi135.stdout: 2024-03-29T11:24:24.361 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:24.362 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.379+0000 7f52b6c82640 10 librados: watch_flush enter 2024-03-29T11:24:24.362 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.379+0000 7f52b6c82640 10 librados: watch_flush exit 2024-03-29T11:24:24.362 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.379+0000 7f52b6c82640 20 client.4186.objecter shutdown clearing up homeless session... 2024-03-29T11:24:24.362 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.379+0000 7f52b6c82640 10 client.4186.objecter successfully canceled tick 2024-03-29T11:24:24.362 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.379+0000 7f52b6c82640 10 monclient: shutdown 2024-03-29T11:24:24.362 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.379+0000 7f52b6c82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:24.362 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.379+0000 7f52b6c82640 1 -- 172.21.15.135:0/1610067259 >> v1:172.21.15.194:6789/0 conn(0x7f52b01408b0 legacy=0x7f52b01d70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.363 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.379+0000 7f52b6c82640 1 -- 172.21.15.135:0/1610067259 shutdown_connections 2024-03-29T11:24:24.363 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.379+0000 7f52b6c82640 1 -- 172.21.15.135:0/1610067259 >> 172.21.15.135:0/1610067259 conn(0x7f52b0066b00 msgr2=0x7f52b013a230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:24.363 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.379+0000 7f52b6c82640 1 -- 172.21.15.135:0/1610067259 shutdown_connections 2024-03-29T11:24:24.363 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.379+0000 7f52b6c82640 1 -- 172.21.15.135:0/1610067259 wait complete. 2024-03-29T11:24:24.363 INFO:teuthology.orchestra.run.smithi135.stderr:2024-03-29T11:24:24.379+0000 7f52b6c82640 1 librados: shutdown 2024-03-29T11:24:24.372 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:24:24.372 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:24:24.393 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-29T11:24:24.411+0000 7fe5b562c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:24:24.464 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:24.482+0000 7f992383e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:24:24.506 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-29T11:24:24.524+0000 7fe5b562c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:24:24.560 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:24.578+0000 7f992383e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:24:24.600 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-29T11:24:24.619+0000 7fe5b562c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:24:24.712 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:24.730+0000 7f992383e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:24:24.752 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-29T11:24:24.771+0000 7fe5b562c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:24:24.755 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:24.773+0000 7f992383e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:24:24.758 INFO:tasks.ceph.osd.6.smithi194.stderr:2024-03-29T11:24:24.776+0000 7fc442b47640 -1 Falling back to public interface 2024-03-29T11:24:24.759 INFO:tasks.ceph.osd.4.smithi194.stderr:2024-03-29T11:24:24.777+0000 7f6ad8549640 -1 Falling back to public interface 2024-03-29T11:24:24.764 INFO:tasks.ceph.osd.1.smithi135.stderr:2024-03-29T11:24:24.782+0000 7f2253b9a640 -1 Falling back to public interface 2024-03-29T11:24:24.768 INFO:tasks.ceph.osd.3.smithi135.stderr:2024-03-29T11:24:24.786+0000 7fb87ed81640 -1 Falling back to public interface 2024-03-29T11:24:24.773 INFO:tasks.ceph.osd.0.smithi135.stderr:2024-03-29T11:24:24.791+0000 7f0181e13640 -1 Falling back to public interface 2024-03-29T11:24:24.773 INFO:tasks.ceph.osd.2.smithi135.stderr:2024-03-29T11:24:24.792+0000 7f3533415640 -1 Falling back to public interface 2024-03-29T11:24:24.790 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-03-29T11:24:24.808+0000 7f114d664640 -1 Falling back to public interface 2024-03-29T11:24:24.794 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-03-29T11:24:24.812+0000 7fe8469bb640 -1 Falling back to public interface 2024-03-29T11:24:24.796 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-29T11:24:24.815+0000 7fe5b562c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:24:24.910 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:24.928+0000 7f992383e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:24:24.950 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-29T11:24:24.969+0000 7fe5b562c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:24:25.177 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:25.265 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.283+0000 7f1354286640 10 monclient: get_monmap_and_config 2024-03-29T11:24:25.266 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.283+0000 7f1354286640 10 monclient: build_initial_monmap 2024-03-29T11:24:25.266 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.283+0000 7f1354286640 10 monclient: monmap: 2024-03-29T11:24:25.266 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:24:25.266 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:25.266 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:24:25.285320+0000 2024-03-29T11:24:25.266 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:24:25.285320+0000 2024-03-29T11:24:25.266 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:25.266 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:25.266 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:25.266 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:24:25.266 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:24:25.266 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:25.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.284+0000 7f1354286640 1 Processor -- start 2024-03-29T11:24:25.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.284+0000 7f1354286640 1 -- start start 2024-03-29T11:24:25.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.284+0000 7f1354286640 10 monclient: init 2024-03-29T11:24:25.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1354286640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:25.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1354286640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:25.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1354286640 10 monclient(hunting): picked mon.noname-c con 0x7f134c1457f0 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:25.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1354286640 10 monclient(hunting): picked mon.noname-a con 0x7f134c13f650 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:25.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1354286640 10 monclient(hunting): picked mon.noname-b con 0x7f134c0a9d70 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:25.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1354286640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134c0aaef0 con 0x7f134c13f650 2024-03-29T11:24:25.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1354286640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134c145bd0 con 0x7f134c1457f0 2024-03-29T11:24:25.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1354286640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134c145eb0 con 0x7f134c0a9d70 2024-03-29T11:24:25.276 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1354286640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:25.276 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1354286640 10 monclient(hunting): authenticate will time out at 772.654153s 2024-03-29T11:24:25.276 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1352823640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f134c0a9d70 0x7f134c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:35166/0 (socket says 172.21.15.135:35166) 2024-03-29T11:24:25.276 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1352823640 1 -- 172.21.15.135:0/1230256949 learned_addr learned my addr 172.21.15.135:0/1230256949 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:24:25.276 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351821640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f134c13f650 0x7f134c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:40074/0 (socket says 172.21.15.135:40074) 2024-03-29T11:24:25.276 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 1 -- 172.21.15.135:0/1230256949 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2095018745 0 0) 0x7f134c0aaef0 con 0x7f134c13f650 2024-03-29T11:24:25.276 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 10 monclient(hunting): _negotiate 2024-03-29T11:24:25.276 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:25.276 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:25.276 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 10 monclient(hunting): my global_id is 4185 2024-03-29T11:24:25.276 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 1 -- 172.21.15.135:0/1230256949 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1338003520 con 0x7f134c13f650 2024-03-29T11:24:25.276 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 1 -- 172.21.15.135:0/1230256949 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1294891459 0 0) 0x7f134c145eb0 con 0x7f134c0a9d70 2024-03-29T11:24:25.276 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 10 monclient(hunting): _negotiate 2024-03-29T11:24:25.276 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:25.276 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:25.277 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 10 monclient(hunting): my global_id is 4184 2024-03-29T11:24:25.277 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 1 -- 172.21.15.135:0/1230256949 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f134c0aaef0 con 0x7f134c0a9d70 2024-03-29T11:24:25.277 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 1 -- 172.21.15.135:0/1230256949 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567740338 0 0) 0x7f134c145bd0 con 0x7f134c1457f0 2024-03-29T11:24:25.277 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 10 monclient(hunting): _negotiate 2024-03-29T11:24:25.277 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:25.277 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:25.277 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 10 monclient(hunting): my global_id is 4189 2024-03-29T11:24:25.277 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 1 -- 172.21.15.135:0/1230256949 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f134c145eb0 con 0x7f134c1457f0 2024-03-29T11:24:25.277 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 1 -- 172.21.15.135:0/1230256949 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4088286992 0 0) 0x7f1338003520 con 0x7f134c13f650 2024-03-29T11:24:25.277 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 1 -- 172.21.15.135:0/1230256949 >> v1:172.21.15.135:6790/0 conn(0x7f134c0a9d70 legacy=0x7f134c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:25.277 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 1 -- 172.21.15.135:0/1230256949 >> v1:172.21.15.194:6789/0 conn(0x7f134c1457f0 legacy=0x7f134c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:25.277 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 10 monclient: _finish_hunting 0 2024-03-29T11:24:25.277 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 1 monclient: found mon.noname-a 2024-03-29T11:24:25.277 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:25.277 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.135:6789/0 2024-03-29T11:24:25.278 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 1 -- 172.21.15.135:0/1230256949 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134c146340 con 0x7f134c13f650 2024-03-29T11:24:25.278 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 10 monclient: _finish_auth 0 2024-03-29T11:24:25.278 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 10 monclient: _check_auth_tickets 2024-03-29T11:24:25.278 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:25.278 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1354286640 5 monclient: authenticate success, global_id 4185 2024-03-29T11:24:25.278 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1354286640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:25.278 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 1 -- 172.21.15.135:0/1230256949 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f133c001690 con 0x7f134c13f650 2024-03-29T11:24:25.278 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:25.278 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:25.278 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 10 monclient: dump: 2024-03-29T11:24:25.278 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:24:25.278 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:25.278 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:25.278 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:25.278 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:25.279 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:25.279 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:25.279 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:25.279 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:25.279 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:25.279 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1354286640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:25.279 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 1 -- 172.21.15.135:0/1230256949 <== mon.0 v1:172.21.15.135:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f133c0018e0 con 0x7f134c13f650 2024-03-29T11:24:25.279 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:25.279 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 1 -- 172.21.15.135:0/1230256949 <== mon.0 v1:172.21.15.135:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f133c001c70 con 0x7f134c13f650 2024-03-29T11:24:25.279 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:25.279 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:25.279 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1351020640 10 monclient: dump: 2024-03-29T11:24:25.279 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:24:25.279 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:25.279 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:25.279 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:25.280 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:25.280 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:25.280 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:25.280 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:25.280 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:25.280 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:25.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1354286640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:25.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.285+0000 7f1354286640 4 set_mon_vals no callback set 2024-03-29T11:24:25.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.286+0000 7f1354286640 10 monclient: shutdown 2024-03-29T11:24:25.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.286+0000 7f1354286640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:25.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.286+0000 7f1354286640 1 -- 172.21.15.135:0/1230256949 >> v1:172.21.15.135:6789/0 conn(0x7f134c13f650 legacy=0x7f134c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:25.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.286+0000 7f1354286640 1 -- 172.21.15.135:0/1230256949 shutdown_connections 2024-03-29T11:24:25.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.286+0000 7f1354286640 1 -- 172.21.15.135:0/1230256949 >> 172.21.15.135:0/1230256949 conn(0x7f134c066a00 msgr2=0x7f134c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:25.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.286+0000 7f1354286640 1 -- 172.21.15.135:0/1230256949 shutdown_connections 2024-03-29T11:24:25.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.286+0000 7f1354286640 1 -- 172.21.15.135:0/1230256949 wait complete. 2024-03-29T11:24:25.281 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.287+0000 7f1354286640 10 monclient: build_initial_monmap 2024-03-29T11:24:25.281 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.287+0000 7f1354286640 10 monclient: monmap: 2024-03-29T11:24:25.281 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:24:25.281 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:25.281 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:24:25.281 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:24:25.281 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:25.281 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:25.281 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:25.281 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:24:25.281 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:24:25.281 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:25.281 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.287+0000 7f1354286640 1 librados: starting msgr at 2024-03-29T11:24:25.281 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.287+0000 7f1354286640 1 librados: starting objecter 2024-03-29T11:24:25.281 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.287+0000 7f1354286640 1 Processor -- start 2024-03-29T11:24:25.281 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.287+0000 7f1354286640 1 -- start start 2024-03-29T11:24:25.281 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.287+0000 7f1354286640 1 librados: setting wanted keys 2024-03-29T11:24:25.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.287+0000 7f1354286640 1 librados: calling monclient init 2024-03-29T11:24:25.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.287+0000 7f1354286640 10 monclient: init 2024-03-29T11:24:25.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1354286640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:25.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1354286640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:25.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1354286640 10 monclient(hunting): picked mon.noname-a con 0x7f134c0a9d70 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:25.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1354286640 10 monclient(hunting): picked mon.noname-b con 0x7f134c13f650 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:25.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1354286640 10 monclient(hunting): picked mon.noname-c con 0x7f134c1457f0 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:25.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1354286640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134c1da810 con 0x7f134c0a9d70 2024-03-29T11:24:25.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1354286640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134c0ad1b0 con 0x7f134c13f650 2024-03-29T11:24:25.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1354286640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134c1db3b0 con 0x7f134c1457f0 2024-03-29T11:24:25.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1354286640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:25.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1354286640 10 monclient(hunting): authenticate will time out at 772.657127s 2024-03-29T11:24:25.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1352022640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f134c0a9d70 0x7f134c0abb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:40084/0 (socket says 172.21.15.135:40084) 2024-03-29T11:24:25.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1352022640 1 -- 172.21.15.135:0/2238937052 learned_addr learned my addr 172.21.15.135:0/2238937052 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:24:25.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1352823640 1 --1- 172.21.15.135:0/2238937052 >> v1:172.21.15.135:6790/0 conn(0x7f134c1457f0 0x7f134c1e1bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:35174/0 (socket says 172.21.15.135:35174) 2024-03-29T11:24:25.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 client.?.objecter ms_handle_connect 0x7f134c0a9d70 2024-03-29T11:24:25.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:25.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 client.?.objecter ms_handle_connect 0x7f134c1457f0 2024-03-29T11:24:25.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:25.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 client.?.objecter ms_handle_connect 0x7f134c13f650 2024-03-29T11:24:25.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:25.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 1 -- 172.21.15.135:0/2238937052 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2825273803 0 0) 0x7f134c1da810 con 0x7f134c0a9d70 2024-03-29T11:24:25.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:25.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:25.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:25.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:24:25.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 1 -- 172.21.15.135:0/2238937052 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1330004eb0 con 0x7f134c0a9d70 2024-03-29T11:24:25.284 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 1 -- 172.21.15.135:0/2238937052 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3007508505 0 0) 0x7f134c1db3b0 con 0x7f134c1457f0 2024-03-29T11:24:25.284 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:25.284 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:25.284 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:25.284 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:24:25.284 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 1 -- 172.21.15.135:0/2238937052 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f134c1da810 con 0x7f134c1457f0 2024-03-29T11:24:25.284 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 1 -- 172.21.15.135:0/2238937052 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2242649894 0 0) 0x7f134c0ad1b0 con 0x7f134c13f650 2024-03-29T11:24:25.284 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:25.284 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:25.284 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:25.284 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 monclient(hunting): my global_id is 4192 2024-03-29T11:24:25.284 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 1 -- 172.21.15.135:0/2238937052 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f134c1db3b0 con 0x7f134c13f650 2024-03-29T11:24:25.284 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 1 -- 172.21.15.135:0/2238937052 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3017702308 0 0) 0x7f1330004eb0 con 0x7f134c0a9d70 2024-03-29T11:24:25.284 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 1 -- 172.21.15.135:0/2238937052 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f134c0ad1b0 con 0x7f134c0a9d70 2024-03-29T11:24:25.284 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 1 -- 172.21.15.135:0/2238937052 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f1348001970 con 0x7f134c0a9d70 2024-03-29T11:24:25.285 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:25.285 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:25.285 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 monclient(hunting): dump: 2024-03-29T11:24:25.285 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:24:25.285 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:25.285 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:25.285 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:25.285 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:25.285 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:25.285 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:25.285 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:25.285 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:25.285 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:25.285 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:25.286 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 1 -- 172.21.15.135:0/2238937052 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4102263072 0 0) 0x7f134c1da810 con 0x7f134c1457f0 2024-03-29T11:24:25.286 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 1 -- 172.21.15.135:0/2238937052 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1330004eb0 con 0x7f134c1457f0 2024-03-29T11:24:25.286 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 1 -- 172.21.15.135:0/2238937052 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1623788930 0 0) 0x7f134c1db3b0 con 0x7f134c13f650 2024-03-29T11:24:25.286 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 1 -- 172.21.15.135:0/2238937052 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f134c1da810 con 0x7f134c13f650 2024-03-29T11:24:25.286 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 1 -- 172.21.15.135:0/2238937052 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f1344001430 con 0x7f134c1457f0 2024-03-29T11:24:25.286 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:25.286 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:25.286 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 monclient(hunting): dump: 2024-03-29T11:24:25.286 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:24:25.286 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:25.286 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:25.286 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:25.287 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:25.287 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:25.287 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:25.287 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:25.287 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:25.287 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:25.287 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 1 -- 172.21.15.135:0/2238937052 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f133c001a50 con 0x7f134c13f650 2024-03-29T11:24:25.287 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:25.287 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:25.287 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 monclient(hunting): dump: 2024-03-29T11:24:25.287 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:24:25.287 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:25.287 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:25.287 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:25.287 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:25.288 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:25.288 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:25.288 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:25.288 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:25.288 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:25.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 1 -- 172.21.15.135:0/2238937052 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 760430314 0 0) 0x7f134c0ad1b0 con 0x7f134c0a9d70 2024-03-29T11:24:25.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 1 -- 172.21.15.135:0/2238937052 >> v1:172.21.15.135:6790/0 conn(0x7f134c1457f0 legacy=0x7f134c1e1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:25.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 1 -- 172.21.15.135:0/2238937052 >> v1:172.21.15.194:6789/0 conn(0x7f134c13f650 legacy=0x7f134c1daca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:25.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:25.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 1 monclient: found mon.a 2024-03-29T11:24:25.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:25.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:25.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 1 -- 172.21.15.135:0/2238937052 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134c1db540 con 0x7f134c0a9d70 2024-03-29T11:24:25.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:25.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:25.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1342ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:25.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1354286640 5 monclient: authenticate success, global_id 4188 2024-03-29T11:24:25.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1354286640 10 monclient: _renew_subs 2024-03-29T11:24:25.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1354286640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:25.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1354286640 1 -- 172.21.15.135:0/2238937052 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f134c13fda0 con 0x7f134c0a9d70 2024-03-29T11:24:25.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1354286640 10 client.4188.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:25.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1354286640 10 monclient: _renew_subs 2024-03-29T11:24:25.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1354286640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:25.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1354286640 1 -- 172.21.15.135:0/2238937052 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f134c142940 con 0x7f134c0a9d70 2024-03-29T11:24:25.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.288+0000 7f1354286640 1 librados: init done 2024-03-29T11:24:25.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.289+0000 7f1342ffd640 1 -- 172.21.15.135:0/2238937052 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1348001c50 con 0x7f134c0a9d70 2024-03-29T11:24:25.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.289+0000 7f1342ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:25.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.289+0000 7f1342ffd640 1 -- 172.21.15.135:0/2238937052 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f13480038c0 con 0x7f134c0a9d70 2024-03-29T11:24:25.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.289+0000 7f1342ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:25.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.289+0000 7f1342ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:25.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.289+0000 7f1342ffd640 10 monclient: dump: 2024-03-29T11:24:25.290 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:24:25.290 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:25.290 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:25.290 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:25.290 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:25.290 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:25.290 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:25.290 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:25.290 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:25.290 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:25.290 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.289+0000 7f135081f640 4 set_mon_vals no callback set 2024-03-29T11:24:25.290 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.289+0000 7f1342ffd640 1 -- 172.21.15.135:0/2238937052 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1348003e30 con 0x7f134c0a9d70 2024-03-29T11:24:25.291 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.289+0000 7f1342ffd640 1 -- 172.21.15.135:0/2238937052 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f1348006890 con 0x7f134c0a9d70 2024-03-29T11:24:25.291 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.289+0000 7f1342ffd640 10 client.4188.objecter ms_dispatch 0x7f134c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:25.291 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.289+0000 7f1342ffd640 3 client.4188.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:25.291 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.289+0000 7f1342ffd640 3 client.4188.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:25.291 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.289+0000 7f1342ffd640 20 client.4188.objecter dump_active .. 0 homeless 2024-03-29T11:24:25.291 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.289+0000 7f1354286640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:25.291 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.289+0000 7f1354286640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:25.291 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.289+0000 7f1354286640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:25.291 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.289+0000 7f1354286640 1 -- 172.21.15.135:0/2238937052 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1334005080 con 0x7f134c0a9d70 2024-03-29T11:24:25.291 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.291+0000 7f1342ffd640 1 -- 172.21.15.135:0/2238937052 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1348004270 con 0x7f134c0a9d70 2024-03-29T11:24:25.291 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.291+0000 7f1342ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:25.291 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.291+0000 7f1342ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:25.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.353+0000 7f1354286640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:25.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.353+0000 7f1354286640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:25.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.353+0000 7f1354286640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:25.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.353+0000 7f1354286640 1 -- 172.21.15.135:0/2238937052 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1334005330 con 0x7f134c0a9d70 2024-03-29T11:24:25.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.354+0000 7f1342ffd640 1 -- 172.21.15.135:0/2238937052 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f13480057c0 con 0x7f134c0a9d70 2024-03-29T11:24:25.337 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.354+0000 7f1342ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:25.337 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.354+0000 7f1342ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:25.337 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:24:25.337 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:25.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.355+0000 7f1354286640 10 librados: watch_flush enter 2024-03-29T11:24:25.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.355+0000 7f1354286640 10 librados: watch_flush exit 2024-03-29T11:24:25.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.355+0000 7f1354286640 20 client.4188.objecter shutdown clearing up homeless session... 2024-03-29T11:24:25.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.355+0000 7f1354286640 10 client.4188.objecter successfully canceled tick 2024-03-29T11:24:25.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.355+0000 7f1354286640 10 monclient: shutdown 2024-03-29T11:24:25.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.355+0000 7f1354286640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:25.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.355+0000 7f1354286640 1 -- 172.21.15.135:0/2238937052 >> v1:172.21.15.135:6789/0 conn(0x7f134c0a9d70 legacy=0x7f134c0abb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:25.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.355+0000 7f1354286640 1 -- 172.21.15.135:0/2238937052 shutdown_connections 2024-03-29T11:24:25.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.355+0000 7f1354286640 1 -- 172.21.15.135:0/2238937052 >> 172.21.15.135:0/2238937052 conn(0x7f134c066a00 msgr2=0x7f134c13b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:25.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.355+0000 7f1354286640 1 -- 172.21.15.135:0/2238937052 shutdown_connections 2024-03-29T11:24:25.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.355+0000 7f1354286640 1 -- 172.21.15.135:0/2238937052 wait complete. 2024-03-29T11:24:25.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:25.355+0000 7f1354286640 1 librados: shutdown 2024-03-29T11:24:25.346 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:25.737 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:25.755+0000 7f992383e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:24:25.764 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-29T11:24:25.782+0000 7fe5b562c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:24:26.012 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:26.030+0000 7f992383e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:24:26.036 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-29T11:24:26.054+0000 7fe5b562c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:24:26.062 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:26.080+0000 7f992383e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:24:26.086 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-29T11:24:26.104+0000 7fe5b562c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:24:26.179 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:26.196+0000 7f992383e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:24:26.205 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-03-29T11:24:26.223+0000 7fe5b562c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:24:27.192 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:27.210+0000 7f98c4551640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:24:27.192 INFO:tasks.ceph.mgr.y.smithi135.stderr:2024-03-29T11:24:27.210+0000 7f98c4551640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:24:28.236 INFO:tasks.ceph.osd.4.smithi194.stderr:2024-03-29T11:24:28.254+0000 7f6ad8549640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:24:28.243 INFO:tasks.ceph.osd.6.smithi194.stderr:2024-03-29T11:24:28.261+0000 7fc442b47640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:24:28.298 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-03-29T11:24:28.316+0000 7f114d664640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:24:28.302 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-03-29T11:24:28.321+0000 7fe8469bb640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:24:28.307 INFO:tasks.ceph.osd.2.smithi135.stderr:2024-03-29T11:24:28.325+0000 7f3533415640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:24:28.308 INFO:tasks.ceph.osd.0.smithi135.stderr:2024-03-29T11:24:28.326+0000 7f0181e13640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:24:28.312 INFO:tasks.ceph.osd.1.smithi135.stderr:2024-03-29T11:24:28.330+0000 7f2253b9a640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:24:28.315 INFO:tasks.ceph.osd.3.smithi135.stderr:2024-03-29T11:24:28.333+0000 7fb87ed81640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:24:32.155 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:32.301 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.320+0000 7f07b66ba640 10 monclient: get_monmap_and_config 2024-03-29T11:24:32.301 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.320+0000 7f07b66ba640 10 monclient: build_initial_monmap 2024-03-29T11:24:32.302 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.320+0000 7f07b66ba640 10 monclient: monmap: 2024-03-29T11:24:32.302 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:24:32.302 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:32.302 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:24:32.320796+0000 2024-03-29T11:24:32.302 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:24:32.320796+0000 2024-03-29T11:24:32.302 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:32.302 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:32.302 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:32.302 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:24:32.302 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:24:32.302 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:32.303 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.321+0000 7f07b66ba640 1 Processor -- start 2024-03-29T11:24:32.303 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.321+0000 7f07b66ba640 1 -- start start 2024-03-29T11:24:32.303 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.321+0000 7f07b66ba640 10 monclient: init 2024-03-29T11:24:32.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.322+0000 7f07b66ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:32.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.322+0000 7f07b66ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:32.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.322+0000 7f07b66ba640 10 monclient(hunting): picked mon.noname-a con 0x7f07b01422b0 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:32.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.322+0000 7f07b66ba640 10 monclient(hunting): picked mon.noname-b con 0x7f07b0142690 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:32.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.322+0000 7f07b66ba640 10 monclient(hunting): picked mon.noname-c con 0x7f07b00b17d0 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:32.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.322+0000 7f07b66ba640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b0142c80 con 0x7f07b01422b0 2024-03-29T11:24:32.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.322+0000 7f07b66ba640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b00aae40 con 0x7f07b00b17d0 2024-03-29T11:24:32.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.322+0000 7f07b66ba640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b013fbd0 con 0x7f07b0142690 2024-03-29T11:24:32.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.322+0000 7f07b66ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:32.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.322+0000 7f07b66ba640 10 monclient(hunting): authenticate will time out at 779.691016s 2024-03-29T11:24:32.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.323+0000 7f07b4c57640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f07b00b17d0 0x7f07b00b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:45858/0 (socket says 172.21.15.135:45858) 2024-03-29T11:24:32.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.323+0000 7f07b4c57640 1 -- 172.21.15.135:0/2449571856 learned_addr learned my addr 172.21.15.135:0/2449571856 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:24:32.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.323+0000 7f07affff640 1 --1- 172.21.15.135:0/2449571856 >> v1:172.21.15.135:6789/0 conn(0x7f07b01422b0 0x7f07b013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:50300/0 (socket says 172.21.15.135:50300) 2024-03-29T11:24:32.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.323+0000 7f07aeffd640 1 -- 172.21.15.135:0/2449571856 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3788033356 0 0) 0x7f07b00aae40 con 0x7f07b00b17d0 2024-03-29T11:24:32.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.323+0000 7f07aeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:32.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.323+0000 7f07aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:32.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.323+0000 7f07aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:32.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.323+0000 7f07aeffd640 10 monclient(hunting): my global_id is 4234 2024-03-29T11:24:32.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.323+0000 7f07aeffd640 1 -- 172.21.15.135:0/2449571856 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0798003520 con 0x7f07b00b17d0 2024-03-29T11:24:32.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 1 -- 172.21.15.135:0/2449571856 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006027927 0 0) 0x7f07b013fbd0 con 0x7f07b0142690 2024-03-29T11:24:32.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:32.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:32.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:32.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 10 monclient(hunting): my global_id is 4226 2024-03-29T11:24:32.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 1 -- 172.21.15.135:0/2449571856 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07b00aae40 con 0x7f07b0142690 2024-03-29T11:24:32.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 1 -- 172.21.15.135:0/2449571856 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1000092521 0 0) 0x7f07b0142c80 con 0x7f07b01422b0 2024-03-29T11:24:32.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:32.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:32.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:32.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 10 monclient(hunting): my global_id is 4233 2024-03-29T11:24:32.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 1 -- 172.21.15.135:0/2449571856 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07b013fbd0 con 0x7f07b01422b0 2024-03-29T11:24:32.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 1 -- 172.21.15.135:0/2449571856 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 668551250 0 0) 0x7f0798003520 con 0x7f07b00b17d0 2024-03-29T11:24:32.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 1 -- 172.21.15.135:0/2449571856 >> v1:172.21.15.135:6790/0 conn(0x7f07b0142690 legacy=0x7f07b00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:32.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 1 -- 172.21.15.135:0/2449571856 >> v1:172.21.15.135:6789/0 conn(0x7f07b01422b0 legacy=0x7f07b013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:32.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:32.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 1 monclient: found mon.noname-c 2024-03-29T11:24:32.310 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:32.310 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:24:32.310 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 1 -- 172.21.15.135:0/2449571856 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b0151940 con 0x7f07b00b17d0 2024-03-29T11:24:32.310 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:32.310 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:32.310 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:32.310 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 1 -- 172.21.15.135:0/2449571856 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f07a4001690 con 0x7f07b00b17d0 2024-03-29T11:24:32.310 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:32.310 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:32.310 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07aeffd640 10 monclient: dump: 2024-03-29T11:24:32.310 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:24:32.310 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:32.310 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:32.310 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:32.311 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:32.311 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:32.311 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:32.311 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:32.311 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:32.311 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:32.311 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07b66ba640 5 monclient: authenticate success, global_id 4234 2024-03-29T11:24:32.311 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.324+0000 7f07b66ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:32.311 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.325+0000 7f07aeffd640 1 -- 172.21.15.135:0/2449571856 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07a40017f0 con 0x7f07b00b17d0 2024-03-29T11:24:32.311 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.325+0000 7f07aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:32.311 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.325+0000 7f07aeffd640 1 -- 172.21.15.135:0/2449571856 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f07a4001b80 con 0x7f07b00b17d0 2024-03-29T11:24:32.311 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.325+0000 7f07aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:32.311 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.325+0000 7f07aeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:32.312 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.325+0000 7f07aeffd640 10 monclient: dump: 2024-03-29T11:24:32.312 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:24:32.312 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:32.312 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:32.312 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:32.312 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:32.312 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:32.312 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:32.312 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:32.312 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:32.312 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:32.312 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.325+0000 7f07b66ba640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:32.313 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.325+0000 7f07b66ba640 4 set_mon_vals no callback set 2024-03-29T11:24:32.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.326+0000 7f07b66ba640 10 monclient: shutdown 2024-03-29T11:24:32.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.326+0000 7f07b66ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:32.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.326+0000 7f07b66ba640 1 -- 172.21.15.135:0/2449571856 >> v1:172.21.15.194:6789/0 conn(0x7f07b00b17d0 legacy=0x7f07b00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:32.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.327+0000 7f07b66ba640 1 -- 172.21.15.135:0/2449571856 shutdown_connections 2024-03-29T11:24:32.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.327+0000 7f07b66ba640 1 -- 172.21.15.135:0/2449571856 >> 172.21.15.135:0/2449571856 conn(0x7f07b0066a00 msgr2=0x7f07b0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:32.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.327+0000 7f07b66ba640 1 -- 172.21.15.135:0/2449571856 shutdown_connections 2024-03-29T11:24:32.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.327+0000 7f07b66ba640 1 -- 172.21.15.135:0/2449571856 wait complete. 2024-03-29T11:24:32.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.328+0000 7f07b66ba640 10 monclient: build_initial_monmap 2024-03-29T11:24:32.321 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.328+0000 7f07b66ba640 10 monclient: monmap: 2024-03-29T11:24:32.321 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:24:32.321 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:32.321 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:24:32.321 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:24:32.321 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:32.321 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:32.321 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:32.321 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:24:32.321 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:24:32.321 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:32.321 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.328+0000 7f07b66ba640 1 librados: starting msgr at 2024-03-29T11:24:32.321 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.328+0000 7f07b66ba640 1 librados: starting objecter 2024-03-29T11:24:32.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.329+0000 7f07b66ba640 1 Processor -- start 2024-03-29T11:24:32.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.329+0000 7f07b66ba640 1 -- start start 2024-03-29T11:24:32.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.329+0000 7f07b66ba640 1 librados: setting wanted keys 2024-03-29T11:24:32.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.329+0000 7f07b66ba640 1 librados: calling monclient init 2024-03-29T11:24:32.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.329+0000 7f07b66ba640 10 monclient: init 2024-03-29T11:24:32.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.330+0000 7f07b66ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:32.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.330+0000 7f07b66ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:32.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.330+0000 7f07b66ba640 10 monclient(hunting): picked mon.noname-a con 0x7f07b00b17d0 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:32.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.330+0000 7f07b66ba640 10 monclient(hunting): picked mon.noname-c con 0x7f07b01422b0 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:32.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.330+0000 7f07b66ba640 10 monclient(hunting): picked mon.noname-b con 0x7f07b0142690 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:32.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.330+0000 7f07b66ba640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b006ff20 con 0x7f07b00b17d0 2024-03-29T11:24:32.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.330+0000 7f07b66ba640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b00711d0 con 0x7f07b0142690 2024-03-29T11:24:32.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.330+0000 7f07b66ba640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b0071c20 con 0x7f07b01422b0 2024-03-29T11:24:32.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.330+0000 7f07b66ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.330+0000 7f07b66ba640 10 monclient(hunting): authenticate will time out at 779.698927s 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.331+0000 7f07b4c57640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f07b0142690 0x7f07b01db910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:46016/0 (socket says 172.21.15.135:46016) 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.331+0000 7f07b4c57640 1 -- 172.21.15.135:0/2935860493 learned_addr learned my addr 172.21.15.135:0/2935860493 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.331+0000 7f07acff9640 10 client.?.objecter ms_handle_connect 0x7f07b0142690 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.331+0000 7f07acff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.331+0000 7f07acff9640 10 client.?.objecter ms_handle_connect 0x7f07b00b17d0 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.331+0000 7f07acff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.331+0000 7f07acff9640 1 -- 172.21.15.135:0/2935860493 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1240940614 0 0) 0x7f07b00711d0 con 0x7f07b0142690 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.331+0000 7f07acff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.331+0000 7f07acff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.331+0000 7f07acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.331+0000 7f07acff9640 10 monclient(hunting): my global_id is 4237 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.331+0000 7f07acff9640 1 -- 172.21.15.135:0/2935860493 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f079c004eb0 con 0x7f07b0142690 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.332+0000 7f07acff9640 1 -- 172.21.15.135:0/2935860493 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 873729648 0 0) 0x7f079c004eb0 con 0x7f07b0142690 2024-03-29T11:24:32.323 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.332+0000 7f07acff9640 1 -- 172.21.15.135:0/2935860493 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07b00711d0 con 0x7f07b0142690 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.332+0000 7f07acff9640 1 -- 172.21.15.135:0/2935860493 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f07a4001b00 con 0x7f07b0142690 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.332+0000 7f07acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.332+0000 7f07acff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.332+0000 7f07acff9640 10 monclient(hunting): dump: 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:32.324 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.332+0000 7f07acff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.332+0000 7f07acff9640 1 -- 172.21.15.135:0/2935860493 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1891735713 0 0) 0x7f07b00711d0 con 0x7f07b0142690 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.332+0000 7f07acff9640 1 -- 172.21.15.135:0/2935860493 >> v1:172.21.15.135:6790/0 conn(0x7f07b01422b0 legacy=0x7f07b013be10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.332+0000 7f07acff9640 1 -- 172.21.15.135:0/2935860493 >> v1:172.21.15.135:6789/0 conn(0x7f07b00b17d0 legacy=0x7f07b0070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.332+0000 7f07acff9640 10 monclient: _finish_hunting 0 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.332+0000 7f07acff9640 1 monclient: found mon.b 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.332+0000 7f07acff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.332+0000 7f07acff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.332+0000 7f07acff9640 1 -- 172.21.15.135:0/2935860493 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b014e5d0 con 0x7f07b0142690 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.333+0000 7f07acff9640 10 monclient: _finish_auth 0 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.333+0000 7f07acff9640 10 monclient: _check_auth_tickets 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.333+0000 7f07acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.333+0000 7f07b66ba640 5 monclient: authenticate success, global_id 4237 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.333+0000 7f07acff9640 1 -- 172.21.15.135:0/2935860493 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07a4004a90 con 0x7f07b0142690 2024-03-29T11:24:32.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.333+0000 7f07b66ba640 10 monclient: _renew_subs 2024-03-29T11:24:32.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.333+0000 7f07b66ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:32.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.333+0000 7f07b66ba640 1 -- 172.21.15.135:0/2935860493 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b013c520 con 0x7f07b0142690 2024-03-29T11:24:32.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.333+0000 7f07acff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:32.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.333+0000 7f07ae7fc640 4 set_mon_vals no callback set 2024-03-29T11:24:32.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.333+0000 7f07b66ba640 10 client.4237.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:32.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.333+0000 7f07b66ba640 10 monclient: _renew_subs 2024-03-29T11:24:32.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.333+0000 7f07b66ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:32.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.333+0000 7f07b66ba640 1 -- 172.21.15.135:0/2935860493 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b01dc200 con 0x7f07b0142690 2024-03-29T11:24:32.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.333+0000 7f07b66ba640 1 librados: init done 2024-03-29T11:24:32.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.334+0000 7f07acff9640 1 -- 172.21.15.135:0/2935860493 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f07a4004da0 con 0x7f07b0142690 2024-03-29T11:24:32.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.334+0000 7f07acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:32.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.334+0000 7f07acff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:32.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.334+0000 7f07acff9640 10 monclient: dump: 2024-03-29T11:24:32.326 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:24:32.327 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:32.327 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:32.327 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:32.327 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:32.327 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:32.327 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:32.327 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:32.327 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:32.327 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:32.327 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.335+0000 7f07b66ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:32.327 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.336+0000 7f07acff9640 1 -- 172.21.15.135:0/2935860493 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f07a401ad70 con 0x7f07b0142690 2024-03-29T11:24:32.327 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.335+0000 7f07b66ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:32.327 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.336+0000 7f07b66ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:32.327 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.336+0000 7f07b66ba640 1 -- 172.21.15.135:0/2935860493 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b0071360 con 0x7f07b0142690 2024-03-29T11:24:32.327 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.337+0000 7f07affff640 10 monclient: get_auth_request con 0x7f079c06dbd0 auth_method 0 2024-03-29T11:24:32.328 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.338+0000 7f07acff9640 1 -- 172.21.15.135:0/2935860493 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f07a4084b90 con 0x7f07b0142690 2024-03-29T11:24:32.328 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.338+0000 7f07acff9640 10 client.4237.objecter ms_dispatch 0x7f07b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:32.328 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.338+0000 7f07acff9640 3 client.4237.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:32.328 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.338+0000 7f07acff9640 3 client.4237.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:32.328 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.338+0000 7f07acff9640 20 client.4237.objecter dump_active .. 0 homeless 2024-03-29T11:24:32.328 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.338+0000 7f07acff9640 10 client.4237.objecter ms_handle_connect 0x7f079c06dbd0 2024-03-29T11:24:32.329 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.343+0000 7f07acff9640 1 -- 172.21.15.135:0/2935860493 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f07a4050470 con 0x7f07b0142690 2024-03-29T11:24:32.329 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.343+0000 7f07acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:32.329 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.343+0000 7f07acff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:32.704 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.722+0000 7f07b66ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:32.704 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.722+0000 7f07b66ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:32.704 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.722+0000 7f07b66ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:24:32.704 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.722+0000 7f07b66ba640 1 -- 172.21.15.135:0/2935860493 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07b014e760 con 0x7f07b0142690 2024-03-29T11:24:32.705 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.723+0000 7f07acff9640 1 -- 172.21.15.135:0/2935860493 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f07a40523a0 con 0x7f07b0142690 2024-03-29T11:24:32.705 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.723+0000 7f07acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:32.705 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.723+0000 7f07acff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:32.705 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:24:32.705 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:32.708 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.727+0000 7f07b66ba640 10 librados: watch_flush enter 2024-03-29T11:24:32.709 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.727+0000 7f07b66ba640 10 librados: watch_flush exit 2024-03-29T11:24:32.709 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.727+0000 7f07b66ba640 20 client.4237.objecter shutdown clearing up homeless session... 2024-03-29T11:24:32.709 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.727+0000 7f07b66ba640 10 client.4237.objecter successfully canceled tick 2024-03-29T11:24:32.709 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.727+0000 7f07b66ba640 1 -- 172.21.15.135:0/2935860493 >> v1:172.21.15.135:6816/3469875285 conn(0x7f079c06dbd0 legacy=0x7f079c070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:32.709 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.727+0000 7f07b66ba640 10 monclient: shutdown 2024-03-29T11:24:32.709 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.727+0000 7f07b66ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:32.709 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.727+0000 7f07b66ba640 1 -- 172.21.15.135:0/2935860493 >> v1:172.21.15.194:6789/0 conn(0x7f07b0142690 legacy=0x7f07b01db910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:32.709 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.727+0000 7f07b66ba640 1 -- 172.21.15.135:0/2935860493 shutdown_connections 2024-03-29T11:24:32.709 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.727+0000 7f07b66ba640 1 -- 172.21.15.135:0/2935860493 >> 172.21.15.135:0/2935860493 conn(0x7f07b0066a00 msgr2=0x7f07b013ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:32.709 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.727+0000 7f07b66ba640 1 -- 172.21.15.135:0/2935860493 shutdown_connections 2024-03-29T11:24:32.710 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.728+0000 7f07b66ba640 1 -- 172.21.15.135:0/2935860493 wait complete. 2024-03-29T11:24:32.710 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:32.728+0000 7f07b66ba640 1 librados: shutdown 2024-03-29T11:24:32.724 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:39.530 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:39.685 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.703+0000 7fc5b7cc1640 10 monclient: get_monmap_and_config 2024-03-29T11:24:39.685 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.703+0000 7fc5b7cc1640 10 monclient: build_initial_monmap 2024-03-29T11:24:39.685 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.703+0000 7fc5b7cc1640 10 monclient: monmap: 2024-03-29T11:24:39.685 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:24:39.685 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:39.685 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:24:39.704153+0000 2024-03-29T11:24:39.685 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:24:39.704153+0000 2024-03-29T11:24:39.685 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:39.685 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:39.685 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:39.686 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:24:39.686 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:24:39.686 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:39.686 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.704+0000 7fc5b7cc1640 1 Processor -- start 2024-03-29T11:24:39.686 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.704+0000 7fc5b7cc1640 1 -- start start 2024-03-29T11:24:39.686 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.704+0000 7fc5b7cc1640 10 monclient: init 2024-03-29T11:24:39.687 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.705+0000 7fc5b7cc1640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:39.687 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.705+0000 7fc5b7cc1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:39.687 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.705+0000 7fc5b7cc1640 10 monclient(hunting): picked mon.noname-a con 0x7fc5b01457f0 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:39.687 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.705+0000 7fc5b7cc1640 10 monclient(hunting): picked mon.noname-c con 0x7fc5b013f650 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:39.687 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.705+0000 7fc5b7cc1640 10 monclient(hunting): picked mon.noname-b con 0x7fc5b00a9d70 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:39.687 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.705+0000 7fc5b7cc1640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b00aaef0 con 0x7fc5b01457f0 2024-03-29T11:24:39.688 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.705+0000 7fc5b7cc1640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b0145bd0 con 0x7fc5b013f650 2024-03-29T11:24:39.688 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.705+0000 7fc5b7cc1640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b0145eb0 con 0x7fc5b00a9d70 2024-03-29T11:24:39.688 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.705+0000 7fc5b625e640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7fc5b00a9d70 0x7fc5b014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:56976/0 (socket says 172.21.15.135:56976) 2024-03-29T11:24:39.688 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.705+0000 7fc5b625e640 1 -- 172.21.15.135:0/3395276241 learned_addr learned my addr 172.21.15.135:0/3395276241 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:24:39.688 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.705+0000 7fc5b7cc1640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:39.688 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.705+0000 7fc5b7cc1640 10 monclient(hunting): authenticate will time out at 787.073827s 2024-03-29T11:24:39.688 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.706+0000 7fc5b4a5b640 1 -- 172.21.15.135:0/3395276241 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431092733 0 0) 0x7fc5b0145eb0 con 0x7fc5b00a9d70 2024-03-29T11:24:39.688 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.706+0000 7fc5b4a5b640 10 monclient(hunting): _negotiate 2024-03-29T11:24:39.688 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.706+0000 7fc5b4a5b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:39.688 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.706+0000 7fc5b4a5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:39.688 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.706+0000 7fc5b4a5b640 10 monclient(hunting): my global_id is 4229 2024-03-29T11:24:39.688 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.706+0000 7fc5b4a5b640 1 -- 172.21.15.135:0/3395276241 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc59c003520 con 0x7fc5b00a9d70 2024-03-29T11:24:39.690 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.706+0000 7fc5b4a5b640 1 -- 172.21.15.135:0/3395276241 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3774053526 0 0) 0x7fc5b0145bd0 con 0x7fc5b013f650 2024-03-29T11:24:39.690 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.706+0000 7fc5b4a5b640 10 monclient(hunting): _negotiate 2024-03-29T11:24:39.690 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.706+0000 7fc5b4a5b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:39.690 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.706+0000 7fc5b4a5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:39.690 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.706+0000 7fc5b4a5b640 10 monclient(hunting): my global_id is 4240 2024-03-29T11:24:39.690 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.706+0000 7fc5b4a5b640 1 -- 172.21.15.135:0/3395276241 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b0145eb0 con 0x7fc5b013f650 2024-03-29T11:24:39.690 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.706+0000 7fc5b4a5b640 1 -- 172.21.15.135:0/3395276241 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042830483 0 0) 0x7fc5b00aaef0 con 0x7fc5b01457f0 2024-03-29T11:24:39.690 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.706+0000 7fc5b4a5b640 10 monclient(hunting): _negotiate 2024-03-29T11:24:39.690 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.706+0000 7fc5b4a5b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:39.690 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.706+0000 7fc5b4a5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:39.691 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.706+0000 7fc5b4a5b640 10 monclient(hunting): my global_id is 4239 2024-03-29T11:24:39.691 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.706+0000 7fc5b4a5b640 1 -- 172.21.15.135:0/3395276241 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b0145bd0 con 0x7fc5b01457f0 2024-03-29T11:24:39.691 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.706+0000 7fc5b4a5b640 1 -- 172.21.15.135:0/3395276241 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 945047626 0 0) 0x7fc59c003520 con 0x7fc5b00a9d70 2024-03-29T11:24:39.691 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.707+0000 7fc5b4a5b640 1 -- 172.21.15.135:0/3395276241 >> v1:172.21.15.194:6789/0 conn(0x7fc5b013f650 legacy=0x7fc5b00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:39.691 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.707+0000 7fc5b4a5b640 1 -- 172.21.15.135:0/3395276241 >> v1:172.21.15.135:6789/0 conn(0x7fc5b01457f0 legacy=0x7fc5b013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:39.691 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.707+0000 7fc5b4a5b640 10 monclient: _finish_hunting 0 2024-03-29T11:24:39.691 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.707+0000 7fc5b4a5b640 1 monclient: found mon.noname-b 2024-03-29T11:24:39.691 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.707+0000 7fc5b4a5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:39.691 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.707+0000 7fc5b4a5b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:24:39.691 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.707+0000 7fc5b4a5b640 1 -- 172.21.15.135:0/3395276241 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b0146340 con 0x7fc5b00a9d70 2024-03-29T11:24:39.691 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.707+0000 7fc5b4a5b640 10 monclient: _finish_auth 0 2024-03-29T11:24:39.691 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.707+0000 7fc5b4a5b640 10 monclient: _check_auth_tickets 2024-03-29T11:24:39.691 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.707+0000 7fc5b4a5b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:39.691 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.707+0000 7fc5b4a5b640 1 -- 172.21.15.135:0/3395276241 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fc5a8001a10 con 0x7fc5b00a9d70 2024-03-29T11:24:39.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.707+0000 7fc5b4a5b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:39.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.707+0000 7fc5b4a5b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:39.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.707+0000 7fc5b4a5b640 10 monclient: dump: 2024-03-29T11:24:39.692 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:24:39.692 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:39.692 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:39.692 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:39.692 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:39.692 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:39.692 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:39.692 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:39.692 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:39.692 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:39.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.707+0000 7fc5b7cc1640 5 monclient: authenticate success, global_id 4229 2024-03-29T11:24:39.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.707+0000 7fc5b7cc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:39.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.707+0000 7fc5b4a5b640 1 -- 172.21.15.135:0/3395276241 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5a8001b70 con 0x7fc5b00a9d70 2024-03-29T11:24:39.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.707+0000 7fc5b4a5b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:39.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.707+0000 7fc5b4a5b640 1 -- 172.21.15.135:0/3395276241 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fc5a8001fc0 con 0x7fc5b00a9d70 2024-03-29T11:24:39.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.707+0000 7fc5b4a5b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:39.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.707+0000 7fc5b4a5b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:39.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.707+0000 7fc5b4a5b640 10 monclient: dump: 2024-03-29T11:24:39.693 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:24:39.693 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:39.693 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:39.693 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:39.693 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:39.693 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:39.693 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:39.693 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:39.693 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:39.694 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:39.694 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.707+0000 7fc5b7cc1640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:39.694 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.707+0000 7fc5b7cc1640 4 set_mon_vals no callback set 2024-03-29T11:24:39.694 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.708+0000 7fc5b7cc1640 10 monclient: shutdown 2024-03-29T11:24:39.694 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.708+0000 7fc5b7cc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:39.694 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.708+0000 7fc5b7cc1640 1 -- 172.21.15.135:0/3395276241 >> v1:172.21.15.135:6790/0 conn(0x7fc5b00a9d70 legacy=0x7fc5b014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:39.694 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.708+0000 7fc5b7cc1640 1 -- 172.21.15.135:0/3395276241 shutdown_connections 2024-03-29T11:24:39.694 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.708+0000 7fc5b7cc1640 1 -- 172.21.15.135:0/3395276241 >> 172.21.15.135:0/3395276241 conn(0x7fc5b0066a00 msgr2=0x7fc5b0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:39.694 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.708+0000 7fc5b7cc1640 1 -- 172.21.15.135:0/3395276241 shutdown_connections 2024-03-29T11:24:39.700 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.708+0000 7fc5b7cc1640 1 -- 172.21.15.135:0/3395276241 wait complete. 2024-03-29T11:24:39.700 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.709+0000 7fc5b7cc1640 10 monclient: build_initial_monmap 2024-03-29T11:24:39.700 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.709+0000 7fc5b7cc1640 10 monclient: monmap: 2024-03-29T11:24:39.700 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:24:39.700 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:39.700 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:24:39.700 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:24:39.700 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:39.701 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:39.701 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:39.701 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:24:39.701 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:24:39.701 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:39.701 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.710+0000 7fc5b7cc1640 1 librados: starting msgr at 2024-03-29T11:24:39.701 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.710+0000 7fc5b7cc1640 1 librados: starting objecter 2024-03-29T11:24:39.701 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.710+0000 7fc5b7cc1640 1 Processor -- start 2024-03-29T11:24:39.701 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.710+0000 7fc5b7cc1640 1 -- start start 2024-03-29T11:24:39.701 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.710+0000 7fc5b7cc1640 1 librados: setting wanted keys 2024-03-29T11:24:39.701 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.710+0000 7fc5b7cc1640 1 librados: calling monclient init 2024-03-29T11:24:39.701 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.710+0000 7fc5b7cc1640 10 monclient: init 2024-03-29T11:24:39.701 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.711+0000 7fc5b7cc1640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:39.701 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.711+0000 7fc5b7cc1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:39.701 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.711+0000 7fc5b7cc1640 10 monclient(hunting): picked mon.noname-c con 0x7fc5b00a9d70 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:39.702 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.711+0000 7fc5b7cc1640 10 monclient(hunting): picked mon.noname-b con 0x7fc5b013f650 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:39.702 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.711+0000 7fc5b7cc1640 10 monclient(hunting): picked mon.noname-a con 0x7fc5b01457f0 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:39.702 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.711+0000 7fc5b7cc1640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b0138cc0 con 0x7fc5b01457f0 2024-03-29T11:24:39.702 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.711+0000 7fc5b525c640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fc5b013f650 0x7fc5b01d74b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:36222/0 (socket says 172.21.15.135:36222) 2024-03-29T11:24:39.702 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.711+0000 7fc5b525c640 1 -- 172.21.15.135:0/1225016528 learned_addr learned my addr 172.21.15.135:0/1225016528 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:24:39.702 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.711+0000 7fc5b7cc1640 1 -- 172.21.15.135:0/1225016528 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b00ad050 con 0x7fc5b013f650 2024-03-29T11:24:39.702 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.711+0000 7fc5a67fc640 10 client.?.objecter ms_handle_connect 0x7fc5b013f650 2024-03-29T11:24:39.702 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.711+0000 7fc5a67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:39.702 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.711+0000 7fc5b7cc1640 1 -- 172.21.15.135:0/1225016528 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b00ad330 con 0x7fc5b00a9d70 2024-03-29T11:24:39.702 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.711+0000 7fc5b7cc1640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:39.703 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.711+0000 7fc5b7cc1640 10 monclient(hunting): authenticate will time out at 787.079878s 2024-03-29T11:24:39.703 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.712+0000 7fc5a67fc640 10 client.?.objecter ms_handle_connect 0x7fc5b00a9d70 2024-03-29T11:24:39.703 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.712+0000 7fc5a67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:39.703 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.712+0000 7fc5a67fc640 1 -- 172.21.15.135:0/1225016528 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061200958 0 0) 0x7fc5b00ad050 con 0x7fc5b013f650 2024-03-29T11:24:39.703 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.712+0000 7fc5a67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:39.703 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.712+0000 7fc5a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:39.703 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.712+0000 7fc5a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:39.703 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.712+0000 7fc5a67fc640 10 monclient(hunting): my global_id is 4243 2024-03-29T11:24:39.703 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.712+0000 7fc5a67fc640 1 -- 172.21.15.135:0/1225016528 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc594004eb0 con 0x7fc5b013f650 2024-03-29T11:24:39.703 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.712+0000 7fc5a67fc640 1 -- 172.21.15.135:0/1225016528 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 812919764 0 0) 0x7fc5b00ad330 con 0x7fc5b00a9d70 2024-03-29T11:24:39.703 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.712+0000 7fc5a67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:39.703 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.712+0000 7fc5a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:39.703 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.712+0000 7fc5a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:39.703 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.712+0000 7fc5a67fc640 10 monclient(hunting): my global_id is 4232 2024-03-29T11:24:39.703 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.712+0000 7fc5a67fc640 1 -- 172.21.15.135:0/1225016528 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b00ad050 con 0x7fc5b00a9d70 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.712+0000 7fc5a67fc640 10 client.?.objecter ms_handle_connect 0x7fc5b01457f0 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.712+0000 7fc5a67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.712+0000 7fc5a67fc640 1 -- 172.21.15.135:0/1225016528 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3280900776 0 0) 0x7fc5b00ad050 con 0x7fc5b00a9d70 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.712+0000 7fc5a67fc640 1 -- 172.21.15.135:0/1225016528 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5b00ad330 con 0x7fc5b00a9d70 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.712+0000 7fc5a67fc640 1 -- 172.21.15.135:0/1225016528 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fc5a0001810 con 0x7fc5b00a9d70 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.712+0000 7fc5a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.712+0000 7fc5a67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.712+0000 7fc5a67fc640 10 monclient(hunting): dump: 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:39.704 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.712+0000 7fc5a67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5a67fc640 1 -- 172.21.15.135:0/1225016528 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618046469 0 0) 0x7fc5b0138cc0 con 0x7fc5b01457f0 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5a67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5a67fc640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5a67fc640 1 -- 172.21.15.135:0/1225016528 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b00ad050 con 0x7fc5b01457f0 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5a67fc640 1 -- 172.21.15.135:0/1225016528 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2345715339 0 0) 0x7fc594004eb0 con 0x7fc5b013f650 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5a67fc640 1 -- 172.21.15.135:0/1225016528 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5b0138cc0 con 0x7fc5b013f650 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5a67fc640 1 -- 172.21.15.135:0/1225016528 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fc5ac001a50 con 0x7fc5b013f650 2024-03-29T11:24:39.705 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5a67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5a67fc640 10 monclient(hunting): dump: 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5a67fc640 1 -- 172.21.15.135:0/1225016528 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1980117693 0 0) 0x7fc5b00ad330 con 0x7fc5b00a9d70 2024-03-29T11:24:39.706 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5a67fc640 1 -- 172.21.15.135:0/1225016528 >> v1:172.21.15.194:6789/0 conn(0x7fc5b013f650 legacy=0x7fc5b01d74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:39.707 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5a67fc640 1 -- 172.21.15.135:0/1225016528 >> v1:172.21.15.135:6789/0 conn(0x7fc5b01457f0 legacy=0x7fc5b01e29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:39.707 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5a67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:39.707 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5a67fc640 1 monclient: found mon.c 2024-03-29T11:24:39.707 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:39.707 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5a67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:24:39.707 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5a67fc640 1 -- 172.21.15.135:0/1225016528 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b00ac1a0 con 0x7fc5b00a9d70 2024-03-29T11:24:39.707 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5a67fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:39.707 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5a67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:39.707 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:39.707 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5b7cc1640 5 monclient: authenticate success, global_id 4232 2024-03-29T11:24:39.707 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5b7cc1640 10 monclient: _renew_subs 2024-03-29T11:24:39.707 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5b7cc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:24:39.707 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5b7cc1640 1 -- 172.21.15.135:0/1225016528 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b00abae0 con 0x7fc5b00a9d70 2024-03-29T11:24:39.707 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5b7cc1640 10 client.4232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:39.707 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5b7cc1640 10 monclient: _renew_subs 2024-03-29T11:24:39.708 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5b7cc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:24:39.708 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5b7cc1640 1 -- 172.21.15.135:0/1225016528 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b01384e0 con 0x7fc5b00a9d70 2024-03-29T11:24:39.708 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.713+0000 7fc5b7cc1640 1 librados: init done 2024-03-29T11:24:39.708 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.714+0000 7fc5a67fc640 1 -- 172.21.15.135:0/1225016528 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5a00019f0 con 0x7fc5b00a9d70 2024-03-29T11:24:39.708 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.714+0000 7fc5a67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:39.708 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.714+0000 7fc5a67fc640 1 -- 172.21.15.135:0/1225016528 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fc5a00037b0 con 0x7fc5b00a9d70 2024-03-29T11:24:39.708 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.714+0000 7fc5a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:39.708 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.714+0000 7fc5a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:39.708 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.714+0000 7fc5a67fc640 10 monclient: dump: 2024-03-29T11:24:39.708 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:24:39.708 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:39.708 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:39.708 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:39.708 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:39.708 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.714+0000 7fc5a7fff640 4 set_mon_vals no callback set 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.714+0000 7fc5b7cc1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.715+0000 7fc5b7cc1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.715+0000 7fc5b7cc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.715+0000 7fc5b7cc1640 1 -- 172.21.15.135:0/1225016528 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b00abf70 con 0x7fc5b00a9d70 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.715+0000 7fc5a67fc640 1 -- 172.21.15.135:0/1225016528 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fc5a00039d0 con 0x7fc5b00a9d70 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.716+0000 7fc5a67fc640 1 -- 172.21.15.135:0/1225016528 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7fc5a0083760 con 0x7fc5b00a9d70 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.716+0000 7fc5a67fc640 10 client.4232.objecter ms_dispatch 0x7fc5b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.716+0000 7fc5a67fc640 3 client.4232.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.716+0000 7fc5a67fc640 3 client.4232.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:39.709 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.716+0000 7fc5b525c640 10 monclient: get_auth_request con 0x7fc59406e000 auth_method 0 2024-03-29T11:24:39.710 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.716+0000 7fc5a67fc640 20 client.4232.objecter dump_active .. 0 homeless 2024-03-29T11:24:39.710 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.716+0000 7fc5a67fc640 10 client.4232.objecter ms_handle_connect 0x7fc59406e000 2024-03-29T11:24:39.710 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.724+0000 7fc5a67fc640 1 -- 172.21.15.135:0/1225016528 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc5a004f040 con 0x7fc5b00a9d70 2024-03-29T11:24:39.710 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.724+0000 7fc5a67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:39.710 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:39.724+0000 7fc5a67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:40.048 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:40.066+0000 7fc5b7cc1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:40.048 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:40.066+0000 7fc5b7cc1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:40.048 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:40.066+0000 7fc5b7cc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:24:40.048 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:40.066+0000 7fc5b7cc1640 1 -- 172.21.15.135:0/1225016528 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5b0138830 con 0x7fc5b00a9d70 2024-03-29T11:24:40.049 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:40.067+0000 7fc5a67fc640 1 -- 172.21.15.135:0/1225016528 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7fc5a0050f70 con 0x7fc5b00a9d70 2024-03-29T11:24:40.049 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:40.067+0000 7fc5a67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:40.049 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:40.067+0000 7fc5a67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:40.049 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:24:40.049 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:40.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:40.071+0000 7fc5b7cc1640 10 librados: watch_flush enter 2024-03-29T11:24:40.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:40.071+0000 7fc5b7cc1640 10 librados: watch_flush exit 2024-03-29T11:24:40.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:40.071+0000 7fc5b7cc1640 20 client.4232.objecter shutdown clearing up homeless session... 2024-03-29T11:24:40.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:40.071+0000 7fc5b7cc1640 10 client.4232.objecter successfully canceled tick 2024-03-29T11:24:40.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:40.071+0000 7fc5b7cc1640 1 -- 172.21.15.135:0/1225016528 >> v1:172.21.15.135:6816/3469875285 conn(0x7fc59406e000 legacy=0x7fc5940704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:40.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:40.071+0000 7fc5b7cc1640 10 monclient: shutdown 2024-03-29T11:24:40.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:40.071+0000 7fc5b7cc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:40.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:40.071+0000 7fc5b7cc1640 1 -- 172.21.15.135:0/1225016528 >> v1:172.21.15.135:6790/0 conn(0x7fc5b00a9d70 legacy=0x7fc5b0137dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:40.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:40.071+0000 7fc5b7cc1640 1 -- 172.21.15.135:0/1225016528 shutdown_connections 2024-03-29T11:24:40.054 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:40.071+0000 7fc5b7cc1640 1 -- 172.21.15.135:0/1225016528 >> 172.21.15.135:0/1225016528 conn(0x7fc5b0066a00 msgr2=0x7fc5b0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:40.054 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:40.071+0000 7fc5b7cc1640 1 -- 172.21.15.135:0/1225016528 shutdown_connections 2024-03-29T11:24:40.054 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:40.071+0000 7fc5b7cc1640 1 -- 172.21.15.135:0/1225016528 wait complete. 2024-03-29T11:24:40.054 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:40.071+0000 7fc5b7cc1640 1 librados: shutdown 2024-03-29T11:24:40.067 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:46.874 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:47.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.042+0000 7fb213fff640 10 monclient: get_monmap_and_config 2024-03-29T11:24:47.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.042+0000 7fb213fff640 10 monclient: build_initial_monmap 2024-03-29T11:24:47.026 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.042+0000 7fb213fff640 10 monclient: monmap: 2024-03-29T11:24:47.026 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:24:47.026 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:47.027 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:24:47.043400+0000 2024-03-29T11:24:47.027 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:24:47.043400+0000 2024-03-29T11:24:47.027 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:47.027 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:47.027 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:47.027 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:24:47.027 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:24:47.027 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:47.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.043+0000 7fb213fff640 1 Processor -- start 2024-03-29T11:24:47.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.043+0000 7fb213fff640 1 -- start start 2024-03-29T11:24:47.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.043+0000 7fb213fff640 10 monclient: init 2024-03-29T11:24:47.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.044+0000 7fb213fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:47.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.044+0000 7fb213fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:47.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.044+0000 7fb213fff640 10 monclient(hunting): picked mon.noname-b con 0x7fb2140a9750 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:47.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.044+0000 7fb213fff640 10 monclient(hunting): picked mon.noname-a con 0x7fb2140add40 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:47.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.044+0000 7fb213fff640 10 monclient(hunting): picked mon.noname-c con 0x7fb2140aad20 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:47.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.045+0000 7fb2137fe640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fb2140aad20 0x7fb214151bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:56534/0 (socket says 172.21.15.135:56534) 2024-03-29T11:24:47.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.045+0000 7fb2137fe640 1 -- 172.21.15.135:0/3440949561 learned_addr learned my addr 172.21.15.135:0/3440949561 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:24:47.038 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.044+0000 7fb213fff640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb214149870 con 0x7fb2140add40 2024-03-29T11:24:47.038 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.045+0000 7fb213fff640 1 -- 172.21.15.135:0/3440949561 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2140abf80 con 0x7fb2140aad20 2024-03-29T11:24:47.038 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.045+0000 7fb213fff640 1 -- 172.21.15.135:0/3440949561 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb214149ac0 con 0x7fb2140a9750 2024-03-29T11:24:47.038 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.045+0000 7fb213fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:47.038 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.045+0000 7fb213fff640 10 monclient(hunting): authenticate will time out at 794.413271s 2024-03-29T11:24:47.038 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.045+0000 7fb211ffb640 1 -- 172.21.15.135:0/3440949561 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3577092600 0 0) 0x7fb2140abf80 con 0x7fb2140aad20 2024-03-29T11:24:47.038 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.045+0000 7fb211ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:47.038 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.045+0000 7fb211ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:47.038 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.045+0000 7fb211ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:47.038 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.045+0000 7fb211ffb640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:24:47.038 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.045+0000 7fb211ffb640 1 -- 172.21.15.135:0/3440949561 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1fc003520 con 0x7fb2140aad20 2024-03-29T11:24:47.038 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.046+0000 7fb211ffb640 1 -- 172.21.15.135:0/3440949561 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 977104378 0 0) 0x7fb1fc003520 con 0x7fb2140aad20 2024-03-29T11:24:47.039 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.046+0000 7fb211ffb640 1 -- 172.21.15.135:0/3440949561 >> v1:172.21.15.135:6790/0 conn(0x7fb2140a9750 legacy=0x7fb214149fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:47.039 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.048+0000 7fb211ffb640 1 -- 172.21.15.135:0/3440949561 >> v1:172.21.15.135:6789/0 conn(0x7fb2140add40 legacy=0x7fb214143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:47.039 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.048+0000 7fb211ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:24:47.039 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.048+0000 7fb211ffb640 1 monclient: found mon.noname-c 2024-03-29T11:24:47.039 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.048+0000 7fb211ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:47.039 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.048+0000 7fb211ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:24:47.039 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.048+0000 7fb211ffb640 1 -- 172.21.15.135:0/3440949561 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb21414a6f0 con 0x7fb2140aad20 2024-03-29T11:24:47.039 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.048+0000 7fb211ffb640 10 monclient: _finish_auth 0 2024-03-29T11:24:47.039 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.048+0000 7fb211ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:24:47.039 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.048+0000 7fb211ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:47.039 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.048+0000 7fb211ffb640 1 -- 172.21.15.135:0/3440949561 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fb208001690 con 0x7fb2140aad20 2024-03-29T11:24:47.039 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.048+0000 7fb211ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:47.039 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.048+0000 7fb211ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:47.039 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.048+0000 7fb211ffb640 10 monclient: dump: 2024-03-29T11:24:47.040 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:24:47.040 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:47.040 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:47.040 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:47.040 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:47.040 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:47.040 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:47.040 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:47.040 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:47.040 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:47.040 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.048+0000 7fb213fff640 5 monclient: authenticate success, global_id 4246 2024-03-29T11:24:47.040 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.048+0000 7fb213fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:47.040 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.048+0000 7fb211ffb640 1 -- 172.21.15.135:0/3440949561 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb208001860 con 0x7fb2140aad20 2024-03-29T11:24:47.041 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.048+0000 7fb211ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:47.041 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.048+0000 7fb211ffb640 1 -- 172.21.15.135:0/3440949561 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fb208001c70 con 0x7fb2140aad20 2024-03-29T11:24:47.041 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.048+0000 7fb211ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:47.041 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.048+0000 7fb211ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:47.041 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.048+0000 7fb211ffb640 10 monclient: dump: 2024-03-29T11:24:47.041 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:24:47.041 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:47.041 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:47.042 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:47.042 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:47.042 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:47.042 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:47.042 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:47.043 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:47.043 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:47.043 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.049+0000 7fb213fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:47.043 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.049+0000 7fb213fff640 4 set_mon_vals no callback set 2024-03-29T11:24:47.043 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.050+0000 7fb213fff640 10 monclient: shutdown 2024-03-29T11:24:47.043 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.050+0000 7fb213fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:47.044 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.050+0000 7fb213fff640 1 -- 172.21.15.135:0/3440949561 >> v1:172.21.15.194:6789/0 conn(0x7fb2140aad20 legacy=0x7fb214151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:47.044 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.050+0000 7fb213fff640 1 -- 172.21.15.135:0/3440949561 shutdown_connections 2024-03-29T11:24:47.044 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.050+0000 7fb213fff640 1 -- 172.21.15.135:0/3440949561 >> 172.21.15.135:0/3440949561 conn(0x7fb2140673e0 msgr2=0x7fb2140b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:47.044 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.050+0000 7fb213fff640 1 -- 172.21.15.135:0/3440949561 shutdown_connections 2024-03-29T11:24:47.045 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.050+0000 7fb213fff640 1 -- 172.21.15.135:0/3440949561 wait complete. 2024-03-29T11:24:47.045 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.052+0000 7fb213fff640 10 monclient: build_initial_monmap 2024-03-29T11:24:47.045 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.052+0000 7fb213fff640 10 monclient: monmap: 2024-03-29T11:24:47.045 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:24:47.045 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:47.045 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:24:47.046 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:24:47.046 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:47.046 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:47.046 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:47.046 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:24:47.046 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:24:47.047 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:47.047 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.052+0000 7fb213fff640 1 librados: starting msgr at 2024-03-29T11:24:47.047 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.052+0000 7fb213fff640 1 librados: starting objecter 2024-03-29T11:24:47.047 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.053+0000 7fb213fff640 1 Processor -- start 2024-03-29T11:24:47.047 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.053+0000 7fb213fff640 1 -- start start 2024-03-29T11:24:47.048 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.053+0000 7fb213fff640 1 librados: setting wanted keys 2024-03-29T11:24:47.048 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.053+0000 7fb213fff640 1 librados: calling monclient init 2024-03-29T11:24:47.048 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.053+0000 7fb213fff640 10 monclient: init 2024-03-29T11:24:47.048 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.054+0000 7fb213fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:47.048 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.054+0000 7fb213fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:47.062 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.054+0000 7fb213fff640 10 monclient(hunting): picked mon.noname-b con 0x7fb2140a9750 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:47.063 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.054+0000 7fb213fff640 10 monclient(hunting): picked mon.noname-c con 0x7fb2140aad20 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:47.063 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.054+0000 7fb213fff640 10 monclient(hunting): picked mon.noname-a con 0x7fb2140add40 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:47.063 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.054+0000 7fb213fff640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb21406f290 con 0x7fb2140add40 2024-03-29T11:24:47.063 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.054+0000 7fb213fff640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb214070240 con 0x7fb2140a9750 2024-03-29T11:24:47.063 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.055+0000 7fb213fff640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb214070520 con 0x7fb2140aad20 2024-03-29T11:24:47.063 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.055+0000 7fb213fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:47.063 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.055+0000 7fb213fff640 10 monclient(hunting): authenticate will time out at 794.423343s 2024-03-29T11:24:47.063 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.058+0000 7fb212ffd640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fb2140a9750 0x7fb21406ea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:56890/0 (socket says 172.21.15.135:56890) 2024-03-29T11:24:47.064 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.058+0000 7fb2137fe640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fb2140add40 0x7fb214071330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:51973/0 (socket says 172.21.15.135:51973) 2024-03-29T11:24:47.064 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.058+0000 7fb2137fe640 1 -- 172.21.15.135:0/3021565087 learned_addr learned my addr 172.21.15.135:0/3021565087 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:24:47.064 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.058+0000 7fb1fb7fe640 10 client.?.objecter ms_handle_connect 0x7fb2140add40 2024-03-29T11:24:47.064 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.058+0000 7fb1fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:47.064 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.059+0000 7fb1fb7fe640 1 -- 172.21.15.135:0/3021565087 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616934435 0 0) 0x7fb21406f290 con 0x7fb2140add40 2024-03-29T11:24:47.064 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.059+0000 7fb1fb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:47.064 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.059+0000 7fb1fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:47.065 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.059+0000 7fb1fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:47.065 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.059+0000 7fb1fb7fe640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:24:47.065 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.059+0000 7fb1fb7fe640 1 -- 172.21.15.135:0/3021565087 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb200004eb0 con 0x7fb2140add40 2024-03-29T11:24:47.065 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.059+0000 7fb1fb7fe640 10 client.?.objecter ms_handle_connect 0x7fb2140a9750 2024-03-29T11:24:47.065 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.059+0000 7fb1fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:47.065 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.059+0000 7fb1fb7fe640 1 -- 172.21.15.135:0/3021565087 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3236514525 0 0) 0x7fb200004eb0 con 0x7fb2140add40 2024-03-29T11:24:47.065 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.059+0000 7fb1fb7fe640 1 -- 172.21.15.135:0/3021565087 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb21406f290 con 0x7fb2140add40 2024-03-29T11:24:47.066 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.059+0000 7fb1fb7fe640 1 -- 172.21.15.135:0/3021565087 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fb208001b70 con 0x7fb2140add40 2024-03-29T11:24:47.066 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.059+0000 7fb1fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:47.066 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.059+0000 7fb1fb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:47.066 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.059+0000 7fb1fb7fe640 10 monclient(hunting): dump: 2024-03-29T11:24:47.066 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:24:47.066 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:47.066 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:47.066 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:47.067 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:47.067 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:47.067 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:47.067 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:47.067 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:47.067 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:47.067 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.059+0000 7fb1fb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:47.068 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.060+0000 7fb1fb7fe640 1 -- 172.21.15.135:0/3021565087 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100718678 0 0) 0x7fb214070240 con 0x7fb2140a9750 2024-03-29T11:24:47.068 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.060+0000 7fb1fb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:47.068 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.060+0000 7fb1fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:47.068 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.060+0000 7fb1fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:47.068 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.060+0000 7fb1fb7fe640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:24:47.068 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.060+0000 7fb1fb7fe640 1 -- 172.21.15.135:0/3021565087 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb200004eb0 con 0x7fb2140a9750 2024-03-29T11:24:47.068 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.061+0000 7fb1fb7fe640 10 client.?.objecter ms_handle_connect 0x7fb2140aad20 2024-03-29T11:24:47.069 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.061+0000 7fb1fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:47.069 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.061+0000 7fb1fb7fe640 1 -- 172.21.15.135:0/3021565087 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 5817256 0 0) 0x7fb200004eb0 con 0x7fb2140a9750 2024-03-29T11:24:47.069 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.061+0000 7fb1fb7fe640 1 -- 172.21.15.135:0/3021565087 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb200005940 con 0x7fb2140a9750 2024-03-29T11:24:47.069 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.061+0000 7fb1fb7fe640 1 -- 172.21.15.135:0/3021565087 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459111914 0 0) 0x7fb214070520 con 0x7fb2140aad20 2024-03-29T11:24:47.069 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.061+0000 7fb1fb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:47.069 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.061+0000 7fb1fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:47.070 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.061+0000 7fb1fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:47.070 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.061+0000 7fb1fb7fe640 10 monclient(hunting): my global_id is 4235 2024-03-29T11:24:47.070 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.061+0000 7fb1fb7fe640 1 -- 172.21.15.135:0/3021565087 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb200005ca0 con 0x7fb2140aad20 2024-03-29T11:24:47.070 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.061+0000 7fb1fb7fe640 1 -- 172.21.15.135:0/3021565087 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fb20c001b20 con 0x7fb2140a9750 2024-03-29T11:24:47.070 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.061+0000 7fb1fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:47.070 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.061+0000 7fb1fb7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:47.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.061+0000 7fb1fb7fe640 10 monclient(hunting): dump: 2024-03-29T11:24:47.071 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:24:47.071 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:47.071 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:47.071 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:47.071 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:47.071 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:47.072 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:47.072 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:47.072 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:47.072 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:47.072 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.062+0000 7fb1fb7fe640 1 -- 172.21.15.135:0/3021565087 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1537226086 0 0) 0x7fb200005ca0 con 0x7fb2140aad20 2024-03-29T11:24:47.072 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.062+0000 7fb1fb7fe640 1 -- 172.21.15.135:0/3021565087 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb214070520 con 0x7fb2140aad20 2024-03-29T11:24:47.072 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.062+0000 7fb1fb7fe640 1 -- 172.21.15.135:0/3021565087 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fb204001740 con 0x7fb2140aad20 2024-03-29T11:24:47.072 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.062+0000 7fb1fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:47.073 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.062+0000 7fb1fb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:47.073 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.062+0000 7fb1fb7fe640 10 monclient(hunting): dump: 2024-03-29T11:24:47.073 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:24:47.073 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:47.073 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:47.073 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:47.073 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:47.073 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:47.074 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:47.074 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:47.074 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:47.074 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:47.074 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.062+0000 7fb1fb7fe640 1 -- 172.21.15.135:0/3021565087 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1632361737 0 0) 0x7fb214070520 con 0x7fb2140aad20 2024-03-29T11:24:47.074 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.062+0000 7fb1fb7fe640 1 -- 172.21.15.135:0/3021565087 >> v1:172.21.15.194:6789/0 conn(0x7fb2140a9750 legacy=0x7fb21406ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:47.075 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.062+0000 7fb1fb7fe640 1 -- 172.21.15.135:0/3021565087 >> v1:172.21.15.135:6789/0 conn(0x7fb2140add40 legacy=0x7fb214071330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:47.075 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.062+0000 7fb1fb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:47.075 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.062+0000 7fb1fb7fe640 1 monclient: found mon.c 2024-03-29T11:24:47.075 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.062+0000 7fb1fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:47.075 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.062+0000 7fb1fb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:24:47.075 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.062+0000 7fb1fb7fe640 1 -- 172.21.15.135:0/3021565087 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb21406e410 con 0x7fb2140aad20 2024-03-29T11:24:47.075 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.062+0000 7fb1fb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:47.076 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.062+0000 7fb1fb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:47.076 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.062+0000 7fb1fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:47.076 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.063+0000 7fb1fb7fe640 1 -- 172.21.15.135:0/3021565087 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb204001c30 con 0x7fb2140aad20 2024-03-29T11:24:47.076 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.063+0000 7fb1fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:47.076 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.063+0000 7fb1fb7fe640 1 -- 172.21.15.135:0/3021565087 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fb2040037b0 con 0x7fb2140aad20 2024-03-29T11:24:47.076 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.063+0000 7fb1fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:47.076 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.063+0000 7fb1fb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:47.076 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.063+0000 7fb1fb7fe640 10 monclient: dump: 2024-03-29T11:24:47.077 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:24:47.077 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:47.077 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:47.077 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:47.077 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:47.077 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:47.077 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:47.078 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:47.078 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:47.078 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:47.078 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.064+0000 7fb213fff640 5 monclient: authenticate success, global_id 4235 2024-03-29T11:24:47.078 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.064+0000 7fb213fff640 10 monclient: _renew_subs 2024-03-29T11:24:47.078 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.064+0000 7fb213fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:24:47.079 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.064+0000 7fb213fff640 1 -- 172.21.15.135:0/3021565087 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb21406e660 con 0x7fb2140aad20 2024-03-29T11:24:47.079 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.064+0000 7fb213fff640 10 client.4235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:47.079 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.064+0000 7fb213fff640 10 monclient: _renew_subs 2024-03-29T11:24:47.079 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.064+0000 7fb213fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:24:47.079 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.064+0000 7fb213fff640 1 -- 172.21.15.135:0/3021565087 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2141e7310 con 0x7fb2140aad20 2024-03-29T11:24:47.079 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.064+0000 7fb213fff640 1 librados: init done 2024-03-29T11:24:47.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.064+0000 7fb2117fa640 4 set_mon_vals no callback set 2024-03-29T11:24:47.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.066+0000 7fb1fb7fe640 1 -- 172.21.15.135:0/3021565087 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fb204003910 con 0x7fb2140aad20 2024-03-29T11:24:47.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.067+0000 7fb213fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:47.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.067+0000 7fb213fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:47.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.067+0000 7fb213fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:24:47.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.067+0000 7fb213fff640 1 -- 172.21.15.135:0/3021565087 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1f4005080 con 0x7fb2140aad20 2024-03-29T11:24:47.081 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.067+0000 7fb1fb7fe640 1 -- 172.21.15.135:0/3021565087 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7fb2040035f0 con 0x7fb2140aad20 2024-03-29T11:24:47.081 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.067+0000 7fb1fb7fe640 10 client.4235.objecter ms_dispatch 0x7fb214000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:47.081 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.067+0000 7fb1fb7fe640 3 client.4235.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:47.081 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.067+0000 7fb1fb7fe640 3 client.4235.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:47.081 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.068+0000 7fb1fb7fe640 20 client.4235.objecter dump_active .. 0 homeless 2024-03-29T11:24:47.081 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.068+0000 7fb212ffd640 10 monclient: get_auth_request con 0x7fb20006e840 auth_method 0 2024-03-29T11:24:47.082 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.069+0000 7fb1fb7fe640 10 client.4235.objecter ms_handle_connect 0x7fb20006e840 2024-03-29T11:24:47.082 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.076+0000 7fb1fb7fe640 1 -- 172.21.15.135:0/3021565087 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb20404f460 con 0x7fb2140aad20 2024-03-29T11:24:47.082 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.076+0000 7fb1fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:47.082 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.076+0000 7fb1fb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:47.382 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.399+0000 7fb213fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:47.382 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.399+0000 7fb213fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:47.382 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.399+0000 7fb213fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:24:47.382 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.399+0000 7fb213fff640 1 -- 172.21.15.135:0/3021565087 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1f4005330 con 0x7fb2140aad20 2024-03-29T11:24:47.382 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.400+0000 7fb1fb7fe640 1 -- 172.21.15.135:0/3021565087 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7fb20404f460 con 0x7fb2140aad20 2024-03-29T11:24:47.383 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.400+0000 7fb1fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:47.383 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.400+0000 7fb1fb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:47.383 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:24:47.383 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:47.385 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.403+0000 7fb213fff640 10 librados: watch_flush enter 2024-03-29T11:24:47.385 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.404+0000 7fb213fff640 10 librados: watch_flush exit 2024-03-29T11:24:47.386 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.404+0000 7fb213fff640 20 client.4235.objecter shutdown clearing up homeless session... 2024-03-29T11:24:47.386 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.404+0000 7fb213fff640 10 client.4235.objecter successfully canceled tick 2024-03-29T11:24:47.386 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.404+0000 7fb213fff640 1 -- 172.21.15.135:0/3021565087 >> v1:172.21.15.135:6816/3469875285 conn(0x7fb20006e840 legacy=0x7fb200070d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:47.386 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.404+0000 7fb213fff640 10 monclient: shutdown 2024-03-29T11:24:47.386 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.404+0000 7fb213fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:47.386 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.404+0000 7fb213fff640 1 -- 172.21.15.135:0/3021565087 >> v1:172.21.15.135:6790/0 conn(0x7fb2140aad20 legacy=0x7fb214070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:47.386 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.404+0000 7fb213fff640 1 -- 172.21.15.135:0/3021565087 shutdown_connections 2024-03-29T11:24:47.387 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.404+0000 7fb213fff640 1 -- 172.21.15.135:0/3021565087 >> 172.21.15.135:0/3021565087 conn(0x7fb2140673e0 msgr2=0x7fb2141406e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:47.387 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.405+0000 7fb213fff640 1 -- 172.21.15.135:0/3021565087 shutdown_connections 2024-03-29T11:24:47.387 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.405+0000 7fb213fff640 1 -- 172.21.15.135:0/3021565087 wait complete. 2024-03-29T11:24:47.388 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:47.406+0000 7fb213fff640 1 librados: shutdown 2024-03-29T11:24:47.406 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:54.216 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:54.370 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.387+0000 7f739248d640 10 monclient: get_monmap_and_config 2024-03-29T11:24:54.371 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.387+0000 7f739248d640 10 monclient: build_initial_monmap 2024-03-29T11:24:54.371 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.387+0000 7f739248d640 10 monclient: monmap: 2024-03-29T11:24:54.371 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:24:54.371 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:54.371 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:24:54.387966+0000 2024-03-29T11:24:54.371 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:24:54.387966+0000 2024-03-29T11:24:54.371 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:54.371 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:54.371 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:54.371 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:24:54.371 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:24:54.371 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:54.372 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.390+0000 7f739248d640 1 Processor -- start 2024-03-29T11:24:54.374 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.391+0000 7f739248d640 1 -- start start 2024-03-29T11:24:54.374 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.392+0000 7f739248d640 10 monclient: init 2024-03-29T11:24:54.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.393+0000 7f739248d640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:54.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.393+0000 7f739248d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:54.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.393+0000 7f739248d640 10 monclient(hunting): picked mon.noname-a con 0x7f738c0ae250 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:54.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.393+0000 7f739248d640 10 monclient(hunting): picked mon.noname-c con 0x7f738c13f970 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:54.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.393+0000 7f739248d640 10 monclient(hunting): picked mon.noname-b con 0x7f738c142800 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:54.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.393+0000 7f739248d640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f738c0afe00 con 0x7f738c0ae250 2024-03-29T11:24:54.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.393+0000 7f739248d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f738c142c10 con 0x7f738c13f970 2024-03-29T11:24:54.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.393+0000 7f739248d640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f738c0afb20 con 0x7f738c142800 2024-03-29T11:24:54.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.393+0000 7f739248d640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:54.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.393+0000 7f739248d640 10 monclient(hunting): authenticate will time out at 801.761165s 2024-03-29T11:24:54.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.395+0000 7f7390a2a640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f738c142800 0x7f738c14da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:55935/0 (socket says 172.21.15.135:55935) 2024-03-29T11:24:54.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.396+0000 7f7390a2a640 1 -- 172.21.15.135:0/35105269 learned_addr learned my addr 172.21.15.135:0/35105269 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:24:54.380 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.396+0000 7f738affd640 1 -- 172.21.15.135:0/35105269 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3404075732 0 0) 0x7f738c0afb20 con 0x7f738c142800 2024-03-29T11:24:54.380 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.396+0000 7f738affd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:54.380 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.396+0000 7f738affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:54.380 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.396+0000 7f738affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:54.380 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.397+0000 7f738affd640 10 monclient(hunting): my global_id is 4238 2024-03-29T11:24:54.380 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.397+0000 7f738affd640 1 -- 172.21.15.135:0/35105269 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7374003520 con 0x7f738c142800 2024-03-29T11:24:54.380 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.397+0000 7f738affd640 1 -- 172.21.15.135:0/35105269 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826650452 0 0) 0x7f738c142c10 con 0x7f738c13f970 2024-03-29T11:24:54.380 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.397+0000 7f738affd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:54.380 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.397+0000 7f738affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:54.380 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.397+0000 7f738affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:54.380 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.397+0000 7f738affd640 10 monclient(hunting): my global_id is 4252 2024-03-29T11:24:54.380 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.397+0000 7f738affd640 1 -- 172.21.15.135:0/35105269 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f738c0afb20 con 0x7f738c13f970 2024-03-29T11:24:54.380 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.397+0000 7f738affd640 1 -- 172.21.15.135:0/35105269 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 488441092 0 0) 0x7f738c0afb20 con 0x7f738c13f970 2024-03-29T11:24:54.381 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.397+0000 7f738affd640 1 -- 172.21.15.135:0/35105269 >> v1:172.21.15.135:6790/0 conn(0x7f738c142800 legacy=0x7f738c14da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:54.381 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.397+0000 7f738affd640 1 -- 172.21.15.135:0/35105269 >> v1:172.21.15.135:6789/0 conn(0x7f738c0ae250 legacy=0x7f738c13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:54.381 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.397+0000 7f738affd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:54.381 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.397+0000 7f738affd640 1 monclient: found mon.noname-c 2024-03-29T11:24:54.381 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.397+0000 7f738affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:54.381 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.397+0000 7f738affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:24:54.381 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.397+0000 7f738affd640 1 -- 172.21.15.135:0/35105269 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738c0aab80 con 0x7f738c13f970 2024-03-29T11:24:54.381 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.397+0000 7f738affd640 10 monclient: _finish_auth 0 2024-03-29T11:24:54.381 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.397+0000 7f738affd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:54.381 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.397+0000 7f738affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:54.381 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.397+0000 7f739248d640 5 monclient: authenticate success, global_id 4252 2024-03-29T11:24:54.381 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.397+0000 7f739248d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:54.381 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.397+0000 7f738affd640 1 -- 172.21.15.135:0/35105269 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f737c001670 con 0x7f738c13f970 2024-03-29T11:24:54.381 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.397+0000 7f738affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:54.381 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.397+0000 7f738affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:54.381 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.397+0000 7f738affd640 10 monclient: dump: 2024-03-29T11:24:54.382 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:24:54.382 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:54.382 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:54.382 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:54.382 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:54.382 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:54.382 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:54.382 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:54.382 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:54.382 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:54.382 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.397+0000 7f739248d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:54.383 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.398+0000 7f738affd640 1 -- 172.21.15.135:0/35105269 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f737c0018c0 con 0x7f738c13f970 2024-03-29T11:24:54.383 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.398+0000 7f738affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:54.383 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.398+0000 7f739248d640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:54.383 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.398+0000 7f739248d640 4 set_mon_vals no callback set 2024-03-29T11:24:54.383 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.399+0000 7f739248d640 10 monclient: shutdown 2024-03-29T11:24:54.383 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.399+0000 7f739248d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:54.383 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.399+0000 7f739248d640 1 -- 172.21.15.135:0/35105269 >> v1:172.21.15.194:6789/0 conn(0x7f738c13f970 legacy=0x7f738c1420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:54.384 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.399+0000 7f739248d640 1 -- 172.21.15.135:0/35105269 shutdown_connections 2024-03-29T11:24:54.384 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.399+0000 7f739248d640 1 -- 172.21.15.135:0/35105269 >> 172.21.15.135:0/35105269 conn(0x7f738c0b35e0 msgr2=0x7f738c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:54.384 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.400+0000 7f739248d640 1 -- 172.21.15.135:0/35105269 shutdown_connections 2024-03-29T11:24:54.384 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.401+0000 7f739248d640 1 -- 172.21.15.135:0/35105269 wait complete. 2024-03-29T11:24:54.384 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.402+0000 7f739248d640 10 monclient: build_initial_monmap 2024-03-29T11:24:54.384 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.402+0000 7f739248d640 10 monclient: monmap: 2024-03-29T11:24:54.384 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:24:54.384 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:54.385 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:24:54.385 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:24:54.385 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:54.385 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:54.385 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:24:54.385 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:24:54.385 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:24:54.385 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:54.385 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.402+0000 7f739248d640 1 librados: starting msgr at 2024-03-29T11:24:54.385 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.402+0000 7f739248d640 1 librados: starting objecter 2024-03-29T11:24:54.385 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.402+0000 7f739248d640 1 Processor -- start 2024-03-29T11:24:54.386 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.402+0000 7f739248d640 1 -- start start 2024-03-29T11:24:54.386 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.402+0000 7f739248d640 1 librados: setting wanted keys 2024-03-29T11:24:54.386 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.402+0000 7f739248d640 1 librados: calling monclient init 2024-03-29T11:24:54.386 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.402+0000 7f739248d640 10 monclient: init 2024-03-29T11:24:54.386 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.403+0000 7f739248d640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:54.386 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.403+0000 7f739248d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:54.387 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.403+0000 7f739248d640 10 monclient(hunting): picked mon.noname-a con 0x7f738c13f970 addr v1:172.21.15.135:6789/0 2024-03-29T11:24:54.387 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.403+0000 7f739248d640 10 monclient(hunting): picked mon.noname-b con 0x7f738c142800 addr v1:172.21.15.194:6789/0 2024-03-29T11:24:54.388 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.403+0000 7f739248d640 10 monclient(hunting): picked mon.noname-c con 0x7f738c0ab0a0 addr v1:172.21.15.135:6790/0 2024-03-29T11:24:54.389 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.403+0000 7f739248d640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f738c13fef0 con 0x7f738c13f970 2024-03-29T11:24:54.389 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.403+0000 7f739248d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f738c140080 con 0x7f738c142800 2024-03-29T11:24:54.389 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.403+0000 7f739248d640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f738c1e80c0 con 0x7f738c0ab0a0 2024-03-29T11:24:54.389 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.403+0000 7f739248d640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:54.389 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.403+0000 7f739248d640 10 monclient(hunting): authenticate will time out at 801.771898s 2024-03-29T11:24:54.393 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.410+0000 7f738bfff640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f738c13f970 0x7f738c1e79b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:49959/0 (socket says 172.21.15.135:49959) 2024-03-29T11:24:54.393 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.410+0000 7f738bfff640 1 -- 172.21.15.135:0/3349766081 learned_addr learned my addr 172.21.15.135:0/3349766081 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:24:54.393 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.410+0000 7f7388ff9640 10 client.?.objecter ms_handle_connect 0x7f738c13f970 2024-03-29T11:24:54.393 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.410+0000 7f7388ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:54.393 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.410+0000 7f7388ff9640 1 -- 172.21.15.135:0/3349766081 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1189237334 0 0) 0x7f738c13fef0 con 0x7f738c13f970 2024-03-29T11:24:54.393 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.410+0000 7f7388ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:54.393 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.410+0000 7f7388ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:54.393 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.410+0000 7f7388ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:54.393 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.410+0000 7f7388ff9640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:24:54.393 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.410+0000 7f7388ff9640 1 -- 172.21.15.135:0/3349766081 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7378004eb0 con 0x7f738c13f970 2024-03-29T11:24:54.397 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.411+0000 7f7388ff9640 1 -- 172.21.15.135:0/3349766081 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 303016036 0 0) 0x7f7378004eb0 con 0x7f738c13f970 2024-03-29T11:24:54.397 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.411+0000 7f7388ff9640 1 -- 172.21.15.135:0/3349766081 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f738c13fef0 con 0x7f738c13f970 2024-03-29T11:24:54.397 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.411+0000 7f7388ff9640 1 -- 172.21.15.135:0/3349766081 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f7380001930 con 0x7f738c13f970 2024-03-29T11:24:54.397 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.411+0000 7f7388ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:54.397 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.411+0000 7f7388ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:54.397 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.411+0000 7f7388ff9640 10 monclient(hunting): dump: 2024-03-29T11:24:54.398 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:24:54.398 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:54.398 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:54.398 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:54.398 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:54.398 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:54.398 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:54.398 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:54.398 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:54.398 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:54.398 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.411+0000 7f7388ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:54.398 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.411+0000 7f7388ff9640 1 -- 172.21.15.135:0/3349766081 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2622226021 0 0) 0x7f738c13fef0 con 0x7f738c13f970 2024-03-29T11:24:54.398 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.412+0000 7f7388ff9640 1 -- 172.21.15.135:0/3349766081 >> v1:172.21.15.135:6790/0 conn(0x7f738c0ab0a0 legacy=0x7f738c1d78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:54.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.412+0000 7f7388ff9640 1 -- 172.21.15.135:0/3349766081 >> v1:172.21.15.194:6789/0 conn(0x7f738c142800 legacy=0x7f738c0abcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:54.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.412+0000 7f7388ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:24:54.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.412+0000 7f7388ff9640 1 monclient: found mon.a 2024-03-29T11:24:54.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.412+0000 7f7388ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:54.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.412+0000 7f7388ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:54.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.412+0000 7f7388ff9640 1 -- 172.21.15.135:0/3349766081 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738c1e8250 con 0x7f738c13f970 2024-03-29T11:24:54.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.412+0000 7f7388ff9640 10 monclient: _finish_auth 0 2024-03-29T11:24:54.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.412+0000 7f7388ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:24:54.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.412+0000 7f7388ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:54.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.412+0000 7f739248d640 5 monclient: authenticate success, global_id 4251 2024-03-29T11:24:54.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.412+0000 7f739248d640 10 monclient: _renew_subs 2024-03-29T11:24:54.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.412+0000 7f739248d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:54.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.412+0000 7f739248d640 1 -- 172.21.15.135:0/3349766081 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f738c0ac400 con 0x7f738c13f970 2024-03-29T11:24:54.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.412+0000 7f739248d640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:54.400 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.412+0000 7f739248d640 10 monclient: _renew_subs 2024-03-29T11:24:54.400 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.412+0000 7f739248d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:54.400 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.412+0000 7f739248d640 1 -- 172.21.15.135:0/3349766081 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f738c1d7ff0 con 0x7f738c13f970 2024-03-29T11:24:54.400 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.412+0000 7f739248d640 1 librados: init done 2024-03-29T11:24:54.400 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.413+0000 7f7388ff9640 1 -- 172.21.15.135:0/3349766081 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7380001d40 con 0x7f738c13f970 2024-03-29T11:24:54.400 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.413+0000 7f7388ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:54.400 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.413+0000 7f738a7fc640 4 set_mon_vals no callback set 2024-03-29T11:24:54.400 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.413+0000 7f7388ff9640 1 -- 172.21.15.135:0/3349766081 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f7380003a60 con 0x7f738c13f970 2024-03-29T11:24:54.400 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.413+0000 7f7388ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:54.400 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.413+0000 7f7388ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:54.400 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.413+0000 7f7388ff9640 10 monclient: dump: 2024-03-29T11:24:54.400 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:24:54.400 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:24:54.400 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:54.401 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:24:54.401 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:54.401 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:24:54.401 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:24:54.401 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:24:54.401 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:24:54.401 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:24:54.401 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.414+0000 7f739248d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:54.401 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.414+0000 7f739248d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:54.401 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.414+0000 7f739248d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:54.401 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.414+0000 7f739248d640 1 -- 172.21.15.135:0/3349766081 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f736c005080 con 0x7f738c13f970 2024-03-29T11:24:54.401 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.415+0000 7f7388ff9640 1 -- 172.21.15.135:0/3349766081 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f7380003bc0 con 0x7f738c13f970 2024-03-29T11:24:54.402 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.415+0000 7f738b7fe640 10 monclient: get_auth_request con 0x7f737806daf0 auth_method 0 2024-03-29T11:24:54.402 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.416+0000 7f7388ff9640 10 client.4251.objecter ms_handle_connect 0x7f737806daf0 2024-03-29T11:24:54.402 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.416+0000 7f7388ff9640 1 -- 172.21.15.135:0/3349766081 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f7380084ac0 con 0x7f738c13f970 2024-03-29T11:24:54.402 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.416+0000 7f7388ff9640 10 client.4251.objecter ms_dispatch 0x7f738c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:54.402 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.416+0000 7f7388ff9640 3 client.4251.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:54.402 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.416+0000 7f7388ff9640 3 client.4251.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:54.402 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.416+0000 7f7388ff9640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-29T11:24:54.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.424+0000 7f7388ff9640 1 -- 172.21.15.135:0/3349766081 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f73800503a0 con 0x7f738c13f970 2024-03-29T11:24:54.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.424+0000 7f7388ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:54.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.424+0000 7f7388ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:54.739 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.756+0000 7f739248d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:54.739 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.756+0000 7f739248d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:54.739 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.756+0000 7f739248d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:24:54.739 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.756+0000 7f739248d640 1 -- 172.21.15.135:0/3349766081 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f736c005330 con 0x7f738c13f970 2024-03-29T11:24:54.742 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.760+0000 7f7388ff9640 1 -- 172.21.15.135:0/3349766081 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f73800522d0 con 0x7f738c13f970 2024-03-29T11:24:54.742 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.760+0000 7f7388ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:54.742 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.760+0000 7f7388ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:54.743 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:24:54.743 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:54.747 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.764+0000 7f739248d640 10 librados: watch_flush enter 2024-03-29T11:24:54.747 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.764+0000 7f739248d640 10 librados: watch_flush exit 2024-03-29T11:24:54.747 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.764+0000 7f739248d640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-29T11:24:54.747 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.764+0000 7f739248d640 10 client.4251.objecter successfully canceled tick 2024-03-29T11:24:54.747 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.765+0000 7f739248d640 1 -- 172.21.15.135:0/3349766081 >> v1:172.21.15.135:6816/3469875285 conn(0x7f737806daf0 legacy=0x7f737806ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:54.748 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.765+0000 7f739248d640 10 monclient: shutdown 2024-03-29T11:24:54.748 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.765+0000 7f739248d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:54.748 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.765+0000 7f739248d640 1 -- 172.21.15.135:0/3349766081 >> v1:172.21.15.135:6789/0 conn(0x7f738c13f970 legacy=0x7f738c1e79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:54.748 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.765+0000 7f739248d640 1 -- 172.21.15.135:0/3349766081 shutdown_connections 2024-03-29T11:24:54.748 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.765+0000 7f739248d640 1 -- 172.21.15.135:0/3349766081 >> 172.21.15.135:0/3349766081 conn(0x7f738c0b35e0 msgr2=0x7f738c0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:54.750 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.768+0000 7f739248d640 1 -- 172.21.15.135:0/3349766081 shutdown_connections 2024-03-29T11:24:54.750 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.768+0000 7f739248d640 1 -- 172.21.15.135:0/3349766081 wait complete. 2024-03-29T11:24:54.750 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:24:54.768+0000 7f739248d640 1 librados: shutdown 2024-03-29T11:24:54.770 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:01.576 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:01.727 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.743+0000 7f662359e640 10 monclient: get_monmap_and_config 2024-03-29T11:25:01.728 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.743+0000 7f662359e640 10 monclient: build_initial_monmap 2024-03-29T11:25:01.728 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.743+0000 7f662359e640 10 monclient: monmap: 2024-03-29T11:25:01.728 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:25:01.728 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:01.728 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:25:01.745090+0000 2024-03-29T11:25:01.728 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:25:01.745090+0000 2024-03-29T11:25:01.728 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:01.728 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:01.728 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:25:01.728 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:25:01.728 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:25:01.729 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:01.729 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.744+0000 7f662359e640 1 Processor -- start 2024-03-29T11:25:01.731 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.745+0000 7f662359e640 1 -- start start 2024-03-29T11:25:01.731 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.745+0000 7f662359e640 10 monclient: init 2024-03-29T11:25:01.731 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.746+0000 7f662359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:01.731 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.746+0000 7f662359e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:01.731 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.746+0000 7f662359e640 10 monclient(hunting): picked mon.noname-c con 0x7f66240a9750 addr v1:172.21.15.194:6789/0 2024-03-29T11:25:01.731 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.746+0000 7f662359e640 10 monclient(hunting): picked mon.noname-a con 0x7f66240ae220 addr v1:172.21.15.135:6789/0 2024-03-29T11:25:01.732 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.746+0000 7f662359e640 10 monclient(hunting): picked mon.noname-b con 0x7f662413b9d0 addr v1:172.21.15.135:6790/0 2024-03-29T11:25:01.732 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.746+0000 7f662359e640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6624142490 con 0x7f66240ae220 2024-03-29T11:25:01.732 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.746+0000 7f662359e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66240ac130 con 0x7f66240a9750 2024-03-29T11:25:01.732 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.746+0000 7f662359e640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66240ac410 con 0x7f662413b9d0 2024-03-29T11:25:01.732 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.746+0000 7f662359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:01.732 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.746+0000 7f662359e640 10 monclient(hunting): authenticate will time out at 809.115698s 2024-03-29T11:25:01.735 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.751+0000 7f662259c640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f66240a9750 0x7f6624141d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:52923/0 (socket says 172.21.15.135:52923) 2024-03-29T11:25:01.735 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.751+0000 7f662259c640 1 -- 172.21.15.135:0/2160374284 learned_addr learned my addr 172.21.15.135:0/2160374284 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:25:01.735 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.751+0000 7f662159a640 1 -- 172.21.15.135:0/2160374284 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968288171 0 0) 0x7f66240ac130 con 0x7f66240a9750 2024-03-29T11:25:01.735 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662159a640 10 monclient(hunting): _negotiate 2024-03-29T11:25:01.735 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:01.735 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:01.735 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662159a640 10 monclient(hunting): my global_id is 4255 2024-03-29T11:25:01.735 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662159a640 1 -- 172.21.15.135:0/2160374284 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f660c003520 con 0x7f66240a9750 2024-03-29T11:25:01.737 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662159a640 1 -- 172.21.15.135:0/2160374284 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2223104459 0 0) 0x7f660c003520 con 0x7f66240a9750 2024-03-29T11:25:01.737 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662159a640 1 -- 172.21.15.135:0/2160374284 >> v1:172.21.15.135:6790/0 conn(0x7f662413b9d0 legacy=0x7f6624149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:01.737 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662159a640 1 -- 172.21.15.135:0/2160374284 >> v1:172.21.15.135:6789/0 conn(0x7f66240ae220 legacy=0x7f662413b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:01.737 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662159a640 10 monclient: _finish_hunting 0 2024-03-29T11:25:01.737 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662159a640 1 monclient: found mon.noname-c 2024-03-29T11:25:01.737 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:01.737 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:25:01.737 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662159a640 1 -- 172.21.15.135:0/2160374284 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66240afa40 con 0x7f66240a9750 2024-03-29T11:25:01.737 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662159a640 10 monclient: _finish_auth 0 2024-03-29T11:25:01.738 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662159a640 10 monclient: _check_auth_tickets 2024-03-29T11:25:01.738 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:01.738 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662159a640 1 -- 172.21.15.135:0/2160374284 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f661c001a10 con 0x7f66240a9750 2024-03-29T11:25:01.738 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:01.738 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:01.738 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662159a640 10 monclient: dump: 2024-03-29T11:25:01.738 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:01.738 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:01.738 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:01.738 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:01.738 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:01.738 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:01.738 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:01.739 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:01.739 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:01.739 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:01.739 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662159a640 1 -- 172.21.15.135:0/2160374284 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f661c001c30 con 0x7f66240a9750 2024-03-29T11:25:01.739 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:01.739 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662159a640 1 -- 172.21.15.135:0/2160374284 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f661c002000 con 0x7f66240a9750 2024-03-29T11:25:01.739 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:01.739 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:01.739 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662159a640 10 monclient: dump: 2024-03-29T11:25:01.739 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:01.739 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:01.739 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:01.739 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:01.739 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:01.739 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:01.740 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:01.740 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:01.740 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:01.740 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:01.740 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662359e640 5 monclient: authenticate success, global_id 4255 2024-03-29T11:25:01.740 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:01.740 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.752+0000 7f662359e640 4 set_mon_vals no callback set 2024-03-29T11:25:01.740 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.753+0000 7f662359e640 10 monclient: shutdown 2024-03-29T11:25:01.740 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.753+0000 7f662359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:01.740 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.753+0000 7f662359e640 1 -- 172.21.15.135:0/2160374284 >> v1:172.21.15.194:6789/0 conn(0x7f66240a9750 legacy=0x7f6624141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:01.741 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.754+0000 7f662359e640 1 -- 172.21.15.135:0/2160374284 shutdown_connections 2024-03-29T11:25:01.742 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.754+0000 7f662359e640 1 -- 172.21.15.135:0/2160374284 >> 172.21.15.135:0/2160374284 conn(0x7f66240673e0 msgr2=0x7f66240b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:01.742 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.755+0000 7f662359e640 1 -- 172.21.15.135:0/2160374284 shutdown_connections 2024-03-29T11:25:01.742 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.755+0000 7f662359e640 1 -- 172.21.15.135:0/2160374284 wait complete. 2024-03-29T11:25:01.742 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.756+0000 7f662359e640 10 monclient: build_initial_monmap 2024-03-29T11:25:01.742 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.756+0000 7f662359e640 10 monclient: monmap: 2024-03-29T11:25:01.742 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:25:01.742 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:01.742 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:25:01.742 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:25:01.742 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:01.743 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:01.743 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:25:01.743 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:25:01.743 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:25:01.743 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:01.743 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.757+0000 7f662359e640 1 librados: starting msgr at 2024-03-29T11:25:01.743 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.757+0000 7f662359e640 1 librados: starting objecter 2024-03-29T11:25:01.743 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.757+0000 7f662359e640 1 Processor -- start 2024-03-29T11:25:01.743 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.757+0000 7f662359e640 1 -- start start 2024-03-29T11:25:01.743 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.757+0000 7f662359e640 1 librados: setting wanted keys 2024-03-29T11:25:01.743 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.757+0000 7f662359e640 1 librados: calling monclient init 2024-03-29T11:25:01.743 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.757+0000 7f662359e640 10 monclient: init 2024-03-29T11:25:01.743 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.758+0000 7f662359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:01.743 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.758+0000 7f662359e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:01.748 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.758+0000 7f662359e640 10 monclient(hunting): picked mon.noname-b con 0x7f66240a9750 addr v1:172.21.15.194:6789/0 2024-03-29T11:25:01.748 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.758+0000 7f662359e640 10 monclient(hunting): picked mon.noname-c con 0x7f66240ae220 addr v1:172.21.15.135:6790/0 2024-03-29T11:25:01.748 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.758+0000 7f662359e640 10 monclient(hunting): picked mon.noname-a con 0x7f662413b9d0 addr v1:172.21.15.135:6789/0 2024-03-29T11:25:01.748 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.758+0000 7f662359e640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f662413e200 con 0x7f662413b9d0 2024-03-29T11:25:01.748 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.758+0000 7f662359e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f662413be30 con 0x7f66240a9750 2024-03-29T11:25:01.748 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.758+0000 7f662359e640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f662413bfc0 con 0x7f66240ae220 2024-03-29T11:25:01.749 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.758+0000 7f662359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:01.749 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.758+0000 7f662359e640 10 monclient(hunting): authenticate will time out at 809.127495s 2024-03-29T11:25:01.749 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.762+0000 7f6622d9d640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f662413b9d0 0x7f662412bf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:58379/0 (socket says 172.21.15.135:58379) 2024-03-29T11:25:01.749 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.762+0000 7f6622d9d640 1 -- 172.21.15.135:0/3558097478 learned_addr learned my addr 172.21.15.135:0/3558097478 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:25:01.749 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.762+0000 7f6612ffd640 10 client.?.objecter ms_handle_connect 0x7f662413b9d0 2024-03-29T11:25:01.749 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.762+0000 7f6612ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:01.749 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.762+0000 7f6612ffd640 1 -- 172.21.15.135:0/3558097478 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152432441 0 0) 0x7f662413e200 con 0x7f662413b9d0 2024-03-29T11:25:01.749 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.762+0000 7f6612ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:01.749 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.762+0000 7f6612ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:01.749 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.762+0000 7f6612ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:01.749 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.762+0000 7f6612ffd640 10 monclient(hunting): my global_id is 4254 2024-03-29T11:25:01.749 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.762+0000 7f6612ffd640 1 -- 172.21.15.135:0/3558097478 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6604004eb0 con 0x7f662413b9d0 2024-03-29T11:25:01.749 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.762+0000 7f6612ffd640 1 -- 172.21.15.135:0/3558097478 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3432842792 0 0) 0x7f6604004eb0 con 0x7f662413b9d0 2024-03-29T11:25:01.749 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.762+0000 7f6612ffd640 1 -- 172.21.15.135:0/3558097478 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f662413e200 con 0x7f662413b9d0 2024-03-29T11:25:01.750 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.762+0000 7f6612ffd640 1 -- 172.21.15.135:0/3558097478 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f66140018c0 con 0x7f662413b9d0 2024-03-29T11:25:01.750 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.762+0000 7f6612ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:01.750 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.762+0000 7f6612ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:01.750 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.762+0000 7f6612ffd640 10 monclient(hunting): dump: 2024-03-29T11:25:01.750 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:01.750 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:01.750 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:01.750 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:01.750 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:01.750 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:01.750 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:01.750 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:01.750 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:01.750 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:01.751 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.762+0000 7f6612ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:01.751 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.763+0000 7f6612ffd640 10 client.?.objecter ms_handle_connect 0x7f66240ae220 2024-03-29T11:25:01.751 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.763+0000 7f6612ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:01.751 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.763+0000 7f6612ffd640 1 -- 172.21.15.135:0/3558097478 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2985539900 0 0) 0x7f662413e200 con 0x7f662413b9d0 2024-03-29T11:25:01.751 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.763+0000 7f6612ffd640 1 -- 172.21.15.135:0/3558097478 >> v1:172.21.15.135:6790/0 conn(0x7f66240ae220 legacy=0x7f662412b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:01.751 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.763+0000 7f6612ffd640 1 -- 172.21.15.135:0/3558097478 >> v1:172.21.15.194:6789/0 conn(0x7f66240a9750 legacy=0x7f662413d4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:01.751 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.763+0000 7f6612ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:01.751 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.763+0000 7f6612ffd640 1 monclient: found mon.a 2024-03-29T11:25:01.751 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.763+0000 7f6612ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:01.751 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.763+0000 7f6612ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:25:01.751 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.763+0000 7f6612ffd640 1 -- 172.21.15.135:0/3558097478 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662413dbd0 con 0x7f662413b9d0 2024-03-29T11:25:01.751 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.763+0000 7f6612ffd640 10 monclient: _finish_auth 0 2024-03-29T11:25:01.751 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.763+0000 7f6612ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:01.751 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.763+0000 7f6612ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:01.751 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.763+0000 7f662359e640 5 monclient: authenticate success, global_id 4254 2024-03-29T11:25:01.752 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.763+0000 7f662359e640 10 monclient: _renew_subs 2024-03-29T11:25:01.752 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.763+0000 7f662359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:25:01.752 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.763+0000 7f662359e640 1 -- 172.21.15.135:0/3558097478 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f662413de80 con 0x7f662413b9d0 2024-03-29T11:25:01.752 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.763+0000 7f662359e640 10 client.4254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:01.752 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.763+0000 7f662359e640 10 monclient: _renew_subs 2024-03-29T11:25:01.752 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.763+0000 7f662359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:25:01.752 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.763+0000 7f662359e640 1 -- 172.21.15.135:0/3558097478 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f662413e870 con 0x7f662413b9d0 2024-03-29T11:25:01.752 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.763+0000 7f662359e640 1 librados: init done 2024-03-29T11:25:01.752 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.764+0000 7f6612ffd640 1 -- 172.21.15.135:0/3558097478 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6614001a20 con 0x7f662413b9d0 2024-03-29T11:25:01.752 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.764+0000 7f6612ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:01.752 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.764+0000 7f6612ffd640 1 -- 172.21.15.135:0/3558097478 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f6614003890 con 0x7f662413b9d0 2024-03-29T11:25:01.752 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.764+0000 7f6612ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:01.752 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.764+0000 7f6612ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:01.752 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.764+0000 7f6612ffd640 10 monclient: dump: 2024-03-29T11:25:01.753 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:01.753 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:01.753 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:01.753 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:01.753 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:01.753 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:01.753 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:01.753 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:01.753 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:01.753 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:01.753 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.764+0000 7f6620d99640 4 set_mon_vals no callback set 2024-03-29T11:25:01.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.766+0000 7f6612ffd640 1 -- 172.21.15.135:0/3558097478 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f6614003b10 con 0x7f662413b9d0 2024-03-29T11:25:01.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.766+0000 7f662359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:01.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.766+0000 7f662359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:01.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.767+0000 7f662359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:25:01.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.767+0000 7f662359e640 1 -- 172.21.15.135:0/3558097478 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6608005080 con 0x7f662413b9d0 2024-03-29T11:25:01.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.767+0000 7f6612ffd640 1 -- 172.21.15.135:0/3558097478 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f6614084a00 con 0x7f662413b9d0 2024-03-29T11:25:01.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.767+0000 7f6612ffd640 10 client.4254.objecter ms_dispatch 0x7f6624000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:01.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.767+0000 7f6612ffd640 3 client.4254.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:01.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.767+0000 7f6612ffd640 3 client.4254.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:01.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.767+0000 7f6612ffd640 20 client.4254.objecter dump_active .. 0 homeless 2024-03-29T11:25:01.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.769+0000 7f662259c640 10 monclient: get_auth_request con 0x7f660406db80 auth_method 0 2024-03-29T11:25:01.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.769+0000 7f6612ffd640 10 client.4254.objecter ms_handle_connect 0x7f660406db80 2024-03-29T11:25:01.761 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.777+0000 7f6612ffd640 1 -- 172.21.15.135:0/3558097478 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f66140502e0 con 0x7f662413b9d0 2024-03-29T11:25:01.761 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.777+0000 7f6612ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:01.761 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:01.777+0000 7f6612ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:02.088 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:02.104+0000 7f662359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:02.088 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:02.104+0000 7f662359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:02.088 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:02.104+0000 7f662359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:25:02.088 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:02.104+0000 7f662359e640 1 -- 172.21.15.135:0/3558097478 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6608005330 con 0x7f662413b9d0 2024-03-29T11:25:02.089 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:02.105+0000 7f6612ffd640 1 -- 172.21.15.135:0/3558097478 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f6614052210 con 0x7f662413b9d0 2024-03-29T11:25:02.089 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:02.105+0000 7f6612ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:02.089 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:02.105+0000 7f6612ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:02.089 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:25:02.089 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:02.093 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:02.108+0000 7f662359e640 10 librados: watch_flush enter 2024-03-29T11:25:02.093 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:02.110+0000 7f662359e640 10 librados: watch_flush exit 2024-03-29T11:25:02.093 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:02.110+0000 7f662359e640 20 client.4254.objecter shutdown clearing up homeless session... 2024-03-29T11:25:02.094 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:02.110+0000 7f662359e640 10 client.4254.objecter successfully canceled tick 2024-03-29T11:25:02.094 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:02.110+0000 7f662359e640 1 -- 172.21.15.135:0/3558097478 >> v1:172.21.15.135:6816/3469875285 conn(0x7f660406db80 legacy=0x7f6604070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:02.094 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:02.110+0000 7f662359e640 10 monclient: shutdown 2024-03-29T11:25:02.094 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:02.110+0000 7f662359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:02.094 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:02.110+0000 7f662359e640 1 -- 172.21.15.135:0/3558097478 >> v1:172.21.15.135:6789/0 conn(0x7f662413b9d0 legacy=0x7f662412bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:02.094 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:02.110+0000 7f662359e640 1 -- 172.21.15.135:0/3558097478 shutdown_connections 2024-03-29T11:25:02.095 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:02.110+0000 7f662359e640 1 -- 172.21.15.135:0/3558097478 >> 172.21.15.135:0/3558097478 conn(0x7f66240673e0 msgr2=0x7f66240ad440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:02.095 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:02.111+0000 7f662359e640 1 -- 172.21.15.135:0/3558097478 shutdown_connections 2024-03-29T11:25:02.095 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:02.111+0000 7f662359e640 1 -- 172.21.15.135:0/3558097478 wait complete. 2024-03-29T11:25:02.095 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:02.111+0000 7f662359e640 1 librados: shutdown 2024-03-29T11:25:02.113 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:08.920 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:09.069 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.084+0000 7fa768bde640 10 monclient: get_monmap_and_config 2024-03-29T11:25:09.069 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.084+0000 7fa768bde640 10 monclient: build_initial_monmap 2024-03-29T11:25:09.069 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.084+0000 7fa768bde640 10 monclient: monmap: 2024-03-29T11:25:09.069 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:25:09.069 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:09.069 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:25:09.085783+0000 2024-03-29T11:25:09.069 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:25:09.085783+0000 2024-03-29T11:25:09.069 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:09.069 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:09.069 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:25:09.070 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:25:09.070 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:25:09.070 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:09.070 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.086+0000 7fa768bde640 1 Processor -- start 2024-03-29T11:25:09.070 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.086+0000 7fa768bde640 1 -- start start 2024-03-29T11:25:09.070 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.086+0000 7fa768bde640 10 monclient: init 2024-03-29T11:25:09.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.087+0000 7fa768bde640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:09.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.087+0000 7fa768bde640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:09.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.087+0000 7fa768bde640 10 monclient(hunting): picked mon.noname-b con 0x7fa7640a9750 addr v1:172.21.15.135:6790/0 2024-03-29T11:25:09.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.087+0000 7fa768bde640 10 monclient(hunting): picked mon.noname-c con 0x7fa7640ae220 addr v1:172.21.15.194:6789/0 2024-03-29T11:25:09.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.087+0000 7fa768bde640 10 monclient(hunting): picked mon.noname-a con 0x7fa76413b9d0 addr v1:172.21.15.135:6789/0 2024-03-29T11:25:09.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.087+0000 7fa768bde640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa764142490 con 0x7fa76413b9d0 2024-03-29T11:25:09.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.087+0000 7fa768bde640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7640ac130 con 0x7fa7640ae220 2024-03-29T11:25:09.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.087+0000 7fa768bde640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7640ac410 con 0x7fa7640a9750 2024-03-29T11:25:09.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.087+0000 7fa768bde640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:09.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.087+0000 7fa768bde640 10 monclient(hunting): authenticate will time out at 816.456279s 2024-03-29T11:25:09.074 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.091+0000 7fa763fff640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fa76413b9d0 0x7fa764149930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:45173/0 (socket says 172.21.15.135:45173) 2024-03-29T11:25:09.074 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.091+0000 7fa763fff640 1 -- 172.21.15.135:0/637406923 learned_addr learned my addr 172.21.15.135:0/637406923 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:25:09.075 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.091+0000 7fa762ffd640 1 --1- 172.21.15.135:0/637406923 >> v1:172.21.15.194:6789/0 conn(0x7fa7640ae220 0x7fa76413b2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:59605/0 (socket says 172.21.15.135:59605) 2024-03-29T11:25:09.076 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.092+0000 7fa7627fc640 1 -- 172.21.15.135:0/637406923 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3883341440 0 0) 0x7fa764142490 con 0x7fa76413b9d0 2024-03-29T11:25:09.076 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.092+0000 7fa7627fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:09.076 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.092+0000 7fa7627fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:09.076 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.092+0000 7fa7627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:09.076 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.092+0000 7fa7627fc640 10 monclient(hunting): my global_id is 4257 2024-03-29T11:25:09.076 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.092+0000 7fa7627fc640 1 -- 172.21.15.135:0/637406923 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa74c003520 con 0x7fa76413b9d0 2024-03-29T11:25:09.076 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.092+0000 7fa7627fc640 1 -- 172.21.15.135:0/637406923 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1435373988 0 0) 0x7fa7640ac130 con 0x7fa7640ae220 2024-03-29T11:25:09.076 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.092+0000 7fa7627fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:09.076 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.092+0000 7fa7627fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:09.077 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.092+0000 7fa7627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:09.077 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.092+0000 7fa7627fc640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:25:09.077 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.092+0000 7fa7627fc640 1 -- 172.21.15.135:0/637406923 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa764142490 con 0x7fa7640ae220 2024-03-29T11:25:09.078 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.093+0000 7fa7627fc640 1 -- 172.21.15.135:0/637406923 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2362671919 0 0) 0x7fa764142490 con 0x7fa7640ae220 2024-03-29T11:25:09.078 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.093+0000 7fa7627fc640 1 -- 172.21.15.135:0/637406923 >> v1:172.21.15.135:6790/0 conn(0x7fa7640a9750 legacy=0x7fa764141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:09.078 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.093+0000 7fa7627fc640 1 -- 172.21.15.135:0/637406923 >> v1:172.21.15.135:6789/0 conn(0x7fa76413b9d0 legacy=0x7fa764149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:09.078 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.093+0000 7fa7627fc640 10 monclient: _finish_hunting 0 2024-03-29T11:25:09.078 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.093+0000 7fa7627fc640 1 monclient: found mon.noname-c 2024-03-29T11:25:09.078 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.093+0000 7fa7627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:09.078 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.093+0000 7fa7627fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:25:09.078 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.093+0000 7fa7627fc640 1 -- 172.21.15.135:0/637406923 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7640afa40 con 0x7fa7640ae220 2024-03-29T11:25:09.078 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.093+0000 7fa7627fc640 10 monclient: _finish_auth 0 2024-03-29T11:25:09.078 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.093+0000 7fa7627fc640 10 monclient: _check_auth_tickets 2024-03-29T11:25:09.078 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.093+0000 7fa7627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:09.079 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.093+0000 7fa7627fc640 1 -- 172.21.15.135:0/637406923 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fa7580015a0 con 0x7fa7640ae220 2024-03-29T11:25:09.079 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.093+0000 7fa7627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:09.079 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.093+0000 7fa7627fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:09.079 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.093+0000 7fa7627fc640 10 monclient: dump: 2024-03-29T11:25:09.079 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:09.079 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:09.079 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:09.079 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:09.079 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:09.079 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:09.079 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:09.079 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:09.079 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:09.079 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:09.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.093+0000 7fa7627fc640 1 -- 172.21.15.135:0/637406923 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa758001700 con 0x7fa7640ae220 2024-03-29T11:25:09.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.093+0000 7fa7627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:09.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.093+0000 7fa7627fc640 1 -- 172.21.15.135:0/637406923 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fa758001a90 con 0x7fa7640ae220 2024-03-29T11:25:09.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.093+0000 7fa7627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:09.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.093+0000 7fa7627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:09.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.093+0000 7fa7627fc640 10 monclient: dump: 2024-03-29T11:25:09.080 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:09.080 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:09.080 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:09.080 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:09.080 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:09.080 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:09.080 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:09.081 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:09.081 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:09.081 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:09.081 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.093+0000 7fa768bde640 5 monclient: authenticate success, global_id 4258 2024-03-29T11:25:09.081 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.093+0000 7fa768bde640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:09.081 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.093+0000 7fa768bde640 4 set_mon_vals no callback set 2024-03-29T11:25:09.081 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.097+0000 7fa768bde640 10 monclient: shutdown 2024-03-29T11:25:09.082 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.097+0000 7fa768bde640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:09.082 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.097+0000 7fa768bde640 1 -- 172.21.15.135:0/637406923 >> v1:172.21.15.194:6789/0 conn(0x7fa7640ae220 legacy=0x7fa76413b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:09.082 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.097+0000 7fa768bde640 1 -- 172.21.15.135:0/637406923 shutdown_connections 2024-03-29T11:25:09.082 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.097+0000 7fa768bde640 1 -- 172.21.15.135:0/637406923 >> 172.21.15.135:0/637406923 conn(0x7fa7640673e0 msgr2=0x7fa7640b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:09.084 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.097+0000 7fa768bde640 1 -- 172.21.15.135:0/637406923 shutdown_connections 2024-03-29T11:25:09.085 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.101+0000 7fa768bde640 1 -- 172.21.15.135:0/637406923 wait complete. 2024-03-29T11:25:09.086 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.101+0000 7fa768bde640 10 monclient: build_initial_monmap 2024-03-29T11:25:09.086 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.101+0000 7fa768bde640 10 monclient: monmap: 2024-03-29T11:25:09.086 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:25:09.086 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:09.086 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:25:09.086 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:25:09.086 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:09.086 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:09.086 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:25:09.086 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:25:09.086 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:25:09.086 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:09.086 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.102+0000 7fa768bde640 1 librados: starting msgr at 2024-03-29T11:25:09.087 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.102+0000 7fa768bde640 1 librados: starting objecter 2024-03-29T11:25:09.087 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.102+0000 7fa768bde640 1 Processor -- start 2024-03-29T11:25:09.087 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.102+0000 7fa768bde640 1 -- start start 2024-03-29T11:25:09.087 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.102+0000 7fa768bde640 1 librados: setting wanted keys 2024-03-29T11:25:09.087 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.102+0000 7fa768bde640 1 librados: calling monclient init 2024-03-29T11:25:09.087 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.102+0000 7fa768bde640 10 monclient: init 2024-03-29T11:25:09.091 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.103+0000 7fa768bde640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:09.091 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.103+0000 7fa768bde640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:09.091 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.103+0000 7fa768bde640 10 monclient(hunting): picked mon.noname-b con 0x7fa7640a9750 addr v1:172.21.15.194:6789/0 2024-03-29T11:25:09.092 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.103+0000 7fa768bde640 10 monclient(hunting): picked mon.noname-a con 0x7fa76413b9d0 addr v1:172.21.15.135:6789/0 2024-03-29T11:25:09.092 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.103+0000 7fa768bde640 10 monclient(hunting): picked mon.noname-c con 0x7fa7641d7bd0 addr v1:172.21.15.135:6790/0 2024-03-29T11:25:09.092 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.103+0000 7fa768bde640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7641cc420 con 0x7fa76413b9d0 2024-03-29T11:25:09.092 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.103+0000 7fa768bde640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7641d7190 con 0x7fa7640a9750 2024-03-29T11:25:09.092 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.103+0000 7fa768bde640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7641d7320 con 0x7fa7641d7bd0 2024-03-29T11:25:09.092 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.103+0000 7fa768bde640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:09.092 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.103+0000 7fa768bde640 10 monclient(hunting): authenticate will time out at 816.472844s 2024-03-29T11:25:09.093 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.109+0000 7fa762ffd640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fa76413b9d0 0x7fa7641cbd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:45307/0 (socket says 172.21.15.135:45307) 2024-03-29T11:25:09.093 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.109+0000 7fa762ffd640 1 -- 172.21.15.135:0/4008721718 learned_addr learned my addr 172.21.15.135:0/4008721718 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:25:09.096 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.109+0000 7fa74bfff640 10 client.?.objecter ms_handle_connect 0x7fa76413b9d0 2024-03-29T11:25:09.096 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.109+0000 7fa74bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:09.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.113+0000 7fa74bfff640 10 client.?.objecter ms_handle_connect 0x7fa7640a9750 2024-03-29T11:25:09.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.113+0000 7fa74bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:09.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.113+0000 7fa74bfff640 1 -- 172.21.15.135:0/4008721718 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2078371520 0 0) 0x7fa7641cc420 con 0x7fa76413b9d0 2024-03-29T11:25:09.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.113+0000 7fa74bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:09.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.113+0000 7fa74bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:09.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.113+0000 7fa74bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:09.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.113+0000 7fa74bfff640 10 monclient(hunting): my global_id is 4260 2024-03-29T11:25:09.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.113+0000 7fa74bfff640 1 -- 172.21.15.135:0/4008721718 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa750004eb0 con 0x7fa76413b9d0 2024-03-29T11:25:09.098 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.113+0000 7fa74bfff640 1 -- 172.21.15.135:0/4008721718 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4279267818 0 0) 0x7fa7641d7190 con 0x7fa7640a9750 2024-03-29T11:25:09.098 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.113+0000 7fa74bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:09.098 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.113+0000 7fa74bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:09.098 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.113+0000 7fa74bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:09.098 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.113+0000 7fa74bfff640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:25:09.098 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.113+0000 7fa74bfff640 1 -- 172.21.15.135:0/4008721718 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7641cc420 con 0x7fa7640a9750 2024-03-29T11:25:09.100 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.114+0000 7fa74bfff640 10 client.?.objecter ms_handle_connect 0x7fa7641d7bd0 2024-03-29T11:25:09.100 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.114+0000 7fa74bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:09.100 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.114+0000 7fa74bfff640 1 -- 172.21.15.135:0/4008721718 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1445160998 0 0) 0x7fa750004eb0 con 0x7fa76413b9d0 2024-03-29T11:25:09.100 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 1 -- 172.21.15.135:0/4008721718 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7641d7190 con 0x7fa76413b9d0 2024-03-29T11:25:09.100 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 1 -- 172.21.15.135:0/4008721718 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fa758001a10 con 0x7fa76413b9d0 2024-03-29T11:25:09.101 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:09.101 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:09.101 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 10 monclient(hunting): dump: 2024-03-29T11:25:09.101 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:09.101 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:09.101 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:09.101 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:09.101 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:09.101 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:09.101 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:09.101 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:09.101 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:09.101 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:09.101 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:09.101 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 1 -- 172.21.15.135:0/4008721718 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 783456212 0 0) 0x7fa7641cc420 con 0x7fa7640a9750 2024-03-29T11:25:09.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 1 -- 172.21.15.135:0/4008721718 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa750004eb0 con 0x7fa7640a9750 2024-03-29T11:25:09.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 1 -- 172.21.15.135:0/4008721718 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440699887 0 0) 0x7fa7641d7320 con 0x7fa7641d7bd0 2024-03-29T11:25:09.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:09.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:09.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:09.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:25:09.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 1 -- 172.21.15.135:0/4008721718 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7641cc420 con 0x7fa7641d7bd0 2024-03-29T11:25:09.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 1 -- 172.21.15.135:0/4008721718 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fa75c001890 con 0x7fa7640a9750 2024-03-29T11:25:09.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:09.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:09.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 10 monclient(hunting): dump: 2024-03-29T11:25:09.102 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:09.102 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:09.102 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:09.103 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:09.103 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:09.103 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:09.103 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:09.103 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:09.103 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:09.103 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:09.103 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 1 -- 172.21.15.135:0/4008721718 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3862340519 0 0) 0x7fa7641d7190 con 0x7fa76413b9d0 2024-03-29T11:25:09.103 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 1 -- 172.21.15.135:0/4008721718 >> v1:172.21.15.135:6790/0 conn(0x7fa7641d7bd0 legacy=0x7fa7641328c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:09.103 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 1 -- 172.21.15.135:0/4008721718 >> v1:172.21.15.194:6789/0 conn(0x7fa7640a9750 legacy=0x7fa7641cafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:09.103 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:09.103 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 1 monclient: found mon.a 2024-03-29T11:25:09.103 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:09.103 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:25:09.103 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 1 -- 172.21.15.135:0/4008721718 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7641d74b0 con 0x7fa76413b9d0 2024-03-29T11:25:09.103 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 10 monclient: _finish_auth 0 2024-03-29T11:25:09.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:09.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa74bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:09.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa768bde640 5 monclient: authenticate success, global_id 4260 2024-03-29T11:25:09.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa768bde640 10 monclient: _renew_subs 2024-03-29T11:25:09.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa768bde640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:25:09.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa768bde640 1 -- 172.21.15.135:0/4008721718 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa764132fd0 con 0x7fa76413b9d0 2024-03-29T11:25:09.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa768bde640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:09.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa768bde640 10 monclient: _renew_subs 2024-03-29T11:25:09.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa768bde640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:25:09.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa768bde640 1 -- 172.21.15.135:0/4008721718 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7640b1660 con 0x7fa76413b9d0 2024-03-29T11:25:09.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.115+0000 7fa768bde640 1 librados: init done 2024-03-29T11:25:09.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.116+0000 7fa74bfff640 1 -- 172.21.15.135:0/4008721718 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa758004a70 con 0x7fa76413b9d0 2024-03-29T11:25:09.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.116+0000 7fa74bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:09.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.116+0000 7fa74bfff640 1 -- 172.21.15.135:0/4008721718 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fa758004e40 con 0x7fa76413b9d0 2024-03-29T11:25:09.105 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.116+0000 7fa7617fa640 4 set_mon_vals no callback set 2024-03-29T11:25:09.105 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.116+0000 7fa74bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:09.105 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.116+0000 7fa74bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:09.105 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.116+0000 7fa74bfff640 10 monclient: dump: 2024-03-29T11:25:09.105 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:09.105 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:09.105 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:09.105 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:09.105 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:09.105 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:09.105 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:09.105 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:09.105 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:09.105 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:09.106 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.117+0000 7fa768bde640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:09.106 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.117+0000 7fa768bde640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:09.107 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.117+0000 7fa768bde640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:25:09.107 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.117+0000 7fa768bde640 1 -- 172.21.15.135:0/4008721718 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa744005080 con 0x7fa76413b9d0 2024-03-29T11:25:09.107 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.122+0000 7fa74bfff640 1 -- 172.21.15.135:0/4008721718 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fa758004fa0 con 0x7fa76413b9d0 2024-03-29T11:25:09.107 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.122+0000 7fa74bfff640 1 -- 172.21.15.135:0/4008721718 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7fa758084ab0 con 0x7fa76413b9d0 2024-03-29T11:25:09.107 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.122+0000 7fa7637fe640 10 monclient: get_auth_request con 0x7fa75006dfc0 auth_method 0 2024-03-29T11:25:09.107 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.122+0000 7fa74bfff640 10 client.4260.objecter ms_dispatch 0x7fa764000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:09.107 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.122+0000 7fa74bfff640 3 client.4260.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:09.107 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.122+0000 7fa74bfff640 3 client.4260.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:09.107 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.123+0000 7fa74bfff640 20 client.4260.objecter dump_active .. 0 homeless 2024-03-29T11:25:09.107 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.123+0000 7fa74bfff640 10 client.4260.objecter ms_handle_connect 0x7fa75006dfc0 2024-03-29T11:25:09.117 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.133+0000 7fa74bfff640 1 -- 172.21.15.135:0/4008721718 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa758050390 con 0x7fa76413b9d0 2024-03-29T11:25:09.117 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.133+0000 7fa74bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:09.117 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.133+0000 7fa74bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:09.434 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.450+0000 7fa768bde640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:09.434 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.450+0000 7fa768bde640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:09.435 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.450+0000 7fa768bde640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:25:09.435 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.450+0000 7fa768bde640 1 -- 172.21.15.135:0/4008721718 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa744005330 con 0x7fa76413b9d0 2024-03-29T11:25:09.436 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.452+0000 7fa74bfff640 1 -- 172.21.15.135:0/4008721718 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7fa7580522c0 con 0x7fa76413b9d0 2024-03-29T11:25:09.436 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.452+0000 7fa74bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:09.436 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.452+0000 7fa74bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:09.436 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:25:09.436 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:09.439 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.456+0000 7fa768bde640 10 librados: watch_flush enter 2024-03-29T11:25:09.440 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.456+0000 7fa768bde640 10 librados: watch_flush exit 2024-03-29T11:25:09.440 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.456+0000 7fa768bde640 20 client.4260.objecter shutdown clearing up homeless session... 2024-03-29T11:25:09.440 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.456+0000 7fa768bde640 10 client.4260.objecter successfully canceled tick 2024-03-29T11:25:09.441 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.457+0000 7fa768bde640 1 -- 172.21.15.135:0/4008721718 >> v1:172.21.15.135:6816/3469875285 conn(0x7fa75006dfc0 legacy=0x7fa750070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:09.441 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.457+0000 7fa768bde640 10 monclient: shutdown 2024-03-29T11:25:09.441 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.457+0000 7fa768bde640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:09.441 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.457+0000 7fa768bde640 1 -- 172.21.15.135:0/4008721718 >> v1:172.21.15.135:6789/0 conn(0x7fa76413b9d0 legacy=0x7fa7641cbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:09.441 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.457+0000 7fa768bde640 1 -- 172.21.15.135:0/4008721718 shutdown_connections 2024-03-29T11:25:09.442 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.457+0000 7fa768bde640 1 -- 172.21.15.135:0/4008721718 >> 172.21.15.135:0/4008721718 conn(0x7fa7640673e0 msgr2=0x7fa7640b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:09.442 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.457+0000 7fa768bde640 1 -- 172.21.15.135:0/4008721718 shutdown_connections 2024-03-29T11:25:09.443 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.460+0000 7fa768bde640 1 -- 172.21.15.135:0/4008721718 wait complete. 2024-03-29T11:25:09.443 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:09.460+0000 7fa768bde640 1 librados: shutdown 2024-03-29T11:25:09.462 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:16.270 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:16.428 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.442+0000 7fa3ef2a8640 10 monclient: get_monmap_and_config 2024-03-29T11:25:16.428 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.442+0000 7fa3ef2a8640 10 monclient: build_initial_monmap 2024-03-29T11:25:16.428 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.442+0000 7fa3ef2a8640 10 monclient: monmap: 2024-03-29T11:25:16.428 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:25:16.428 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:16.429 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:25:16.443677+0000 2024-03-29T11:25:16.429 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:25:16.443677+0000 2024-03-29T11:25:16.429 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:16.429 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:16.429 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:25:16.429 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:25:16.429 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:25:16.429 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:16.430 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.444+0000 7fa3ef2a8640 1 Processor -- start 2024-03-29T11:25:16.430 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.444+0000 7fa3ef2a8640 1 -- start start 2024-03-29T11:25:16.430 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.444+0000 7fa3ef2a8640 10 monclient: init 2024-03-29T11:25:16.430 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.445+0000 7fa3ef2a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:16.430 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.445+0000 7fa3ef2a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:16.430 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.445+0000 7fa3ef2a8640 10 monclient(hunting): picked mon.noname-c con 0x7fa3e80a9930 addr v1:172.21.15.194:6789/0 2024-03-29T11:25:16.430 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.445+0000 7fa3ef2a8640 10 monclient(hunting): picked mon.noname-b con 0x7fa3e8149dc0 addr v1:172.21.15.135:6790/0 2024-03-29T11:25:16.430 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.445+0000 7fa3ef2a8640 10 monclient(hunting): picked mon.noname-a con 0x7fa3e814ff40 addr v1:172.21.15.135:6789/0 2024-03-29T11:25:16.430 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.445+0000 7fa3ef2a8640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e80aae60 con 0x7fa3e814ff40 2024-03-29T11:25:16.430 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.445+0000 7fa3ef2a8640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e80b4310 con 0x7fa3e80a9930 2024-03-29T11:25:16.430 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.445+0000 7fa3ef2a8640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e80afdd0 con 0x7fa3e8149dc0 2024-03-29T11:25:16.430 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.445+0000 7fa3ef2a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:16.431 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.445+0000 7fa3ef2a8640 10 monclient(hunting): authenticate will time out at 823.814854s 2024-03-29T11:25:16.433 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.449+0000 7fa3ed845640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fa3e814ff40 0x7fa3e8152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:60277/0 (socket says 172.21.15.135:60277) 2024-03-29T11:25:16.433 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.449+0000 7fa3ed845640 1 -- 172.21.15.135:0/2466441266 learned_addr learned my addr 172.21.15.135:0/2466441266 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:25:16.435 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.451+0000 7fa3dffff640 1 -- 172.21.15.135:0/2466441266 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082140577 0 0) 0x7fa3e80afdd0 con 0x7fa3e8149dc0 2024-03-29T11:25:16.435 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.451+0000 7fa3dffff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:16.435 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.451+0000 7fa3dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:16.435 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.451+0000 7fa3dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:16.436 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.451+0000 7fa3dffff640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:25:16.436 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.451+0000 7fa3dffff640 1 -- 172.21.15.135:0/2466441266 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3d4003520 con 0x7fa3e8149dc0 2024-03-29T11:25:16.436 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.451+0000 7fa3dffff640 1 -- 172.21.15.135:0/2466441266 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2459646172 0 0) 0x7fa3d4003520 con 0x7fa3e8149dc0 2024-03-29T11:25:16.436 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.451+0000 7fa3dffff640 1 -- 172.21.15.135:0/2466441266 >> v1:172.21.15.194:6789/0 conn(0x7fa3e80a9930 legacy=0x7fa3e8143a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:16.438 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.453+0000 7fa3dffff640 1 -- 172.21.15.135:0/2466441266 >> v1:172.21.15.135:6789/0 conn(0x7fa3e814ff40 legacy=0x7fa3e8152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:16.438 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.453+0000 7fa3dffff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:16.438 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.453+0000 7fa3dffff640 1 monclient: found mon.noname-b 2024-03-29T11:25:16.438 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.453+0000 7fa3dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:16.438 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.453+0000 7fa3dffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:25:16.438 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.453+0000 7fa3dffff640 1 -- 172.21.15.135:0/2466441266 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e814a950 con 0x7fa3e8149dc0 2024-03-29T11:25:16.438 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.453+0000 7fa3dffff640 10 monclient: _finish_auth 0 2024-03-29T11:25:16.438 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.453+0000 7fa3dffff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:16.438 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.453+0000 7fa3dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:16.438 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.453+0000 7fa3dffff640 1 -- 172.21.15.135:0/2466441266 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fa3e00019f0 con 0x7fa3e8149dc0 2024-03-29T11:25:16.438 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.453+0000 7fa3dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:16.438 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.453+0000 7fa3dffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:16.439 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.453+0000 7fa3dffff640 10 monclient: dump: 2024-03-29T11:25:16.439 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:16.439 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:16.439 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:16.439 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:16.439 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:16.439 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:16.439 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:16.439 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:16.439 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:16.439 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:16.439 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.453+0000 7fa3ef2a8640 5 monclient: authenticate success, global_id 4247 2024-03-29T11:25:16.439 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.453+0000 7fa3ef2a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:16.442 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.457+0000 7fa3dffff640 1 -- 172.21.15.135:0/2466441266 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3e0001c50 con 0x7fa3e8149dc0 2024-03-29T11:25:16.442 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.457+0000 7fa3dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:16.442 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.458+0000 7fa3dffff640 1 -- 172.21.15.135:0/2466441266 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fa3e00020a0 con 0x7fa3e8149dc0 2024-03-29T11:25:16.442 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.458+0000 7fa3dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:16.442 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.458+0000 7fa3dffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:16.442 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.458+0000 7fa3dffff640 10 monclient: dump: 2024-03-29T11:25:16.442 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:16.442 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:16.443 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:16.443 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:16.443 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:16.443 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:16.443 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:16.443 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:16.443 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:16.443 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:16.444 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.458+0000 7fa3ef2a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:16.444 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.458+0000 7fa3ef2a8640 4 set_mon_vals no callback set 2024-03-29T11:25:16.444 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.459+0000 7fa3ef2a8640 10 monclient: shutdown 2024-03-29T11:25:16.444 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.459+0000 7fa3ef2a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:16.444 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.459+0000 7fa3ef2a8640 1 -- 172.21.15.135:0/2466441266 >> v1:172.21.15.135:6790/0 conn(0x7fa3e8149dc0 legacy=0x7fa3e814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:16.444 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.460+0000 7fa3ef2a8640 1 -- 172.21.15.135:0/2466441266 shutdown_connections 2024-03-29T11:25:16.444 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.460+0000 7fa3ef2a8640 1 -- 172.21.15.135:0/2466441266 >> 172.21.15.135:0/2466441266 conn(0x7fa3e80b35e0 msgr2=0x7fa3e80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:16.445 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.461+0000 7fa3ef2a8640 1 -- 172.21.15.135:0/2466441266 shutdown_connections 2024-03-29T11:25:16.445 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.461+0000 7fa3ef2a8640 1 -- 172.21.15.135:0/2466441266 wait complete. 2024-03-29T11:25:16.447 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.462+0000 7fa3ef2a8640 10 monclient: build_initial_monmap 2024-03-29T11:25:16.447 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.462+0000 7fa3ef2a8640 10 monclient: monmap: 2024-03-29T11:25:16.447 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:25:16.447 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:16.447 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:25:16.447 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:25:16.447 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:16.447 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:16.447 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:25:16.447 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:25:16.447 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:25:16.447 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:16.447 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.462+0000 7fa3ef2a8640 1 librados: starting msgr at 2024-03-29T11:25:16.447 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.462+0000 7fa3ef2a8640 1 librados: starting objecter 2024-03-29T11:25:16.448 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.462+0000 7fa3ef2a8640 1 Processor -- start 2024-03-29T11:25:16.449 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.464+0000 7fa3ef2a8640 1 -- start start 2024-03-29T11:25:16.449 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.464+0000 7fa3ef2a8640 1 librados: setting wanted keys 2024-03-29T11:25:16.449 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.464+0000 7fa3ef2a8640 1 librados: calling monclient init 2024-03-29T11:25:16.449 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.464+0000 7fa3ef2a8640 10 monclient: init 2024-03-29T11:25:16.451 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.466+0000 7fa3ef2a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:16.451 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.466+0000 7fa3ef2a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:16.451 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.466+0000 7fa3ef2a8640 10 monclient(hunting): picked mon.noname-a con 0x7fa3e80a9930 addr v1:172.21.15.135:6789/0 2024-03-29T11:25:16.451 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.466+0000 7fa3ef2a8640 10 monclient(hunting): picked mon.noname-b con 0x7fa3e8149dc0 addr v1:172.21.15.194:6789/0 2024-03-29T11:25:16.451 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.466+0000 7fa3ef2a8640 10 monclient(hunting): picked mon.noname-c con 0x7fa3e814ff40 addr v1:172.21.15.135:6790/0 2024-03-29T11:25:16.451 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.466+0000 7fa3ef2a8640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e8148eb0 con 0x7fa3e80a9930 2024-03-29T11:25:16.451 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.466+0000 7fa3ef2a8640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e8149a70 con 0x7fa3e8149dc0 2024-03-29T11:25:16.451 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.466+0000 7fa3ef2a8640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e8127a60 con 0x7fa3e814ff40 2024-03-29T11:25:16.451 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.466+0000 7fa3ef2a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:16.451 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.466+0000 7fa3ef2a8640 10 monclient(hunting): authenticate will time out at 823.835653s 2024-03-29T11:25:16.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.471+0000 7fa3ed044640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fa3e80a9930 0x7fa3e8149360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:60423/0 (socket says 172.21.15.135:60423) 2024-03-29T11:25:16.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.471+0000 7fa3ed044640 1 -- 172.21.15.135:0/2061251472 learned_addr learned my addr 172.21.15.135:0/2061251472 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:25:16.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.471+0000 7fa3ddffb640 10 client.?.objecter ms_handle_connect 0x7fa3e80a9930 2024-03-29T11:25:16.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.471+0000 7fa3ddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:16.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.471+0000 7fa3ddffb640 1 -- 172.21.15.135:0/2061251472 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2094747491 0 0) 0x7fa3e8148eb0 con 0x7fa3e80a9930 2024-03-29T11:25:16.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.471+0000 7fa3ddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:16.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.471+0000 7fa3ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:16.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.471+0000 7fa3ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:16.459 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.471+0000 7fa3ddffb640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:25:16.459 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.471+0000 7fa3ddffb640 1 -- 172.21.15.135:0/2061251472 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3cc004eb0 con 0x7fa3e80a9930 2024-03-29T11:25:16.459 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.471+0000 7fa3ddffb640 1 -- 172.21.15.135:0/2061251472 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3665611125 0 0) 0x7fa3cc004eb0 con 0x7fa3e80a9930 2024-03-29T11:25:16.459 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.471+0000 7fa3ddffb640 1 -- 172.21.15.135:0/2061251472 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3e8148eb0 con 0x7fa3e80a9930 2024-03-29T11:25:16.459 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.471+0000 7fa3ddffb640 1 -- 172.21.15.135:0/2061251472 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fa3d8004080 con 0x7fa3e80a9930 2024-03-29T11:25:16.459 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.471+0000 7fa3ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:16.459 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.471+0000 7fa3ddffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:16.459 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.471+0000 7fa3ddffb640 10 monclient(hunting): dump: 2024-03-29T11:25:16.459 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:16.459 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:16.459 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:16.459 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:16.459 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:16.459 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:16.460 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:16.460 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:16.460 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:16.460 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:16.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.471+0000 7fa3ddffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:16.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.472+0000 7fa3ddffb640 10 client.?.objecter ms_handle_connect 0x7fa3e8149dc0 2024-03-29T11:25:16.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.472+0000 7fa3ddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:16.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.472+0000 7fa3ddffb640 1 -- 172.21.15.135:0/2061251472 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 760164611 0 0) 0x7fa3e8148eb0 con 0x7fa3e80a9930 2024-03-29T11:25:16.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.472+0000 7fa3ddffb640 1 -- 172.21.15.135:0/2061251472 >> v1:172.21.15.135:6790/0 conn(0x7fa3e814ff40 legacy=0x7fa3e81ee9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:16.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.472+0000 7fa3ddffb640 1 -- 172.21.15.135:0/2061251472 >> v1:172.21.15.194:6789/0 conn(0x7fa3e8149dc0 legacy=0x7fa3e80ac820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:16.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.472+0000 7fa3ddffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:16.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.472+0000 7fa3ddffb640 1 monclient: found mon.a 2024-03-29T11:25:16.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.472+0000 7fa3ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:16.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.472+0000 7fa3ddffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:25:16.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.472+0000 7fa3ddffb640 1 -- 172.21.15.135:0/2061251472 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e8127bf0 con 0x7fa3e80a9930 2024-03-29T11:25:16.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.472+0000 7fa3ddffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:16.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.472+0000 7fa3ddffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:16.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.472+0000 7fa3ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:16.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.472+0000 7fa3ef2a8640 5 monclient: authenticate success, global_id 4266 2024-03-29T11:25:16.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.472+0000 7fa3ef2a8640 10 monclient: _renew_subs 2024-03-29T11:25:16.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.472+0000 7fa3ef2a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:25:16.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.472+0000 7fa3ef2a8640 1 -- 172.21.15.135:0/2061251472 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e80ab260 con 0x7fa3e80a9930 2024-03-29T11:25:16.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.472+0000 7fa3ef2a8640 10 client.4266.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:16.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.472+0000 7fa3ef2a8640 10 monclient: _renew_subs 2024-03-29T11:25:16.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.472+0000 7fa3ef2a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:25:16.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.472+0000 7fa3ef2a8640 1 -- 172.21.15.135:0/2061251472 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e80ad110 con 0x7fa3e80a9930 2024-03-29T11:25:16.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.472+0000 7fa3ef2a8640 1 librados: init done 2024-03-29T11:25:16.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.473+0000 7fa3ddffb640 1 -- 172.21.15.135:0/2061251472 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3d80044d0 con 0x7fa3e80a9930 2024-03-29T11:25:16.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.473+0000 7fa3ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:16.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.474+0000 7fa3ddffb640 1 -- 172.21.15.135:0/2061251472 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fa3d80042e0 con 0x7fa3e80a9930 2024-03-29T11:25:16.462 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.474+0000 7fa3ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:16.462 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.474+0000 7fa3ddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:16.462 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.474+0000 7fa3ddffb640 10 monclient: dump: 2024-03-29T11:25:16.462 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:16.462 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:16.462 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:16.462 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:16.462 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:16.462 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:16.462 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:16.462 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:16.462 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:16.462 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:16.462 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.474+0000 7fa3deffd640 4 set_mon_vals no callback set 2024-03-29T11:25:16.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.475+0000 7fa3ef2a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:16.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.475+0000 7fa3ef2a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:16.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.475+0000 7fa3ef2a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:25:16.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.475+0000 7fa3ef2a8640 1 -- 172.21.15.135:0/2061251472 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3c8005080 con 0x7fa3e80a9930 2024-03-29T11:25:16.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.476+0000 7fa3ddffb640 1 -- 172.21.15.135:0/2061251472 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fa3d80013e0 con 0x7fa3e80a9930 2024-03-29T11:25:16.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.477+0000 7fa3ec843640 10 monclient: get_auth_request con 0x7fa3cc06dc00 auth_method 0 2024-03-29T11:25:16.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.477+0000 7fa3ddffb640 10 client.4266.objecter ms_handle_connect 0x7fa3cc06dc00 2024-03-29T11:25:16.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.477+0000 7fa3ddffb640 1 -- 172.21.15.135:0/2061251472 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7fa3d8084af0 con 0x7fa3e80a9930 2024-03-29T11:25:16.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.478+0000 7fa3ddffb640 10 client.4266.objecter ms_dispatch 0x7fa3e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:16.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.478+0000 7fa3ddffb640 3 client.4266.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:16.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.478+0000 7fa3ddffb640 3 client.4266.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:16.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.478+0000 7fa3ddffb640 20 client.4266.objecter dump_active .. 0 homeless 2024-03-29T11:25:16.472 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.488+0000 7fa3ddffb640 1 -- 172.21.15.135:0/2061251472 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa3d80503d0 con 0x7fa3e80a9930 2024-03-29T11:25:16.472 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.488+0000 7fa3ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:16.472 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.488+0000 7fa3ddffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:16.798 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.812+0000 7fa3ef2a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:16.798 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.812+0000 7fa3ef2a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:16.798 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.812+0000 7fa3ef2a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:25:16.798 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.812+0000 7fa3ef2a8640 1 -- 172.21.15.135:0/2061251472 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3c8005330 con 0x7fa3e80a9930 2024-03-29T11:25:16.798 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.813+0000 7fa3ddffb640 1 -- 172.21.15.135:0/2061251472 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7fa3d8052300 con 0x7fa3e80a9930 2024-03-29T11:25:16.798 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.813+0000 7fa3ddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:16.798 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.813+0000 7fa3ddffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:16.801 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:25:16.801 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:16.805 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.818+0000 7fa3ef2a8640 10 librados: watch_flush enter 2024-03-29T11:25:16.806 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.820+0000 7fa3ef2a8640 10 librados: watch_flush exit 2024-03-29T11:25:16.806 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.820+0000 7fa3ef2a8640 20 client.4266.objecter shutdown clearing up homeless session... 2024-03-29T11:25:16.806 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.820+0000 7fa3ef2a8640 10 client.4266.objecter successfully canceled tick 2024-03-29T11:25:16.806 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.820+0000 7fa3ef2a8640 1 -- 172.21.15.135:0/2061251472 >> v1:172.21.15.135:6816/3469875285 conn(0x7fa3cc06dc00 legacy=0x7fa3cc0700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:16.806 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.820+0000 7fa3ef2a8640 10 monclient: shutdown 2024-03-29T11:25:16.806 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.820+0000 7fa3ef2a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:16.806 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.820+0000 7fa3ef2a8640 1 -- 172.21.15.135:0/2061251472 >> v1:172.21.15.135:6789/0 conn(0x7fa3e80a9930 legacy=0x7fa3e8149360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:16.806 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.821+0000 7fa3ef2a8640 1 -- 172.21.15.135:0/2061251472 shutdown_connections 2024-03-29T11:25:16.806 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.821+0000 7fa3ef2a8640 1 -- 172.21.15.135:0/2061251472 >> 172.21.15.135:0/2061251472 conn(0x7fa3e80b35e0 msgr2=0x7fa3e80b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:16.809 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.823+0000 7fa3ef2a8640 1 -- 172.21.15.135:0/2061251472 shutdown_connections 2024-03-29T11:25:16.809 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.823+0000 7fa3ef2a8640 1 -- 172.21.15.135:0/2061251472 wait complete. 2024-03-29T11:25:16.809 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:16.825+0000 7fa3ef2a8640 1 librados: shutdown 2024-03-29T11:25:16.827 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:23.634 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:23.793 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.809+0000 7ff456b32640 10 monclient: get_monmap_and_config 2024-03-29T11:25:23.793 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.809+0000 7ff456b32640 10 monclient: build_initial_monmap 2024-03-29T11:25:23.793 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.809+0000 7ff456b32640 10 monclient: monmap: 2024-03-29T11:25:23.793 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:25:23.793 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:23.793 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:25:23.810363+0000 2024-03-29T11:25:23.793 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:25:23.810363+0000 2024-03-29T11:25:23.793 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:23.793 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:23.794 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:25:23.794 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:25:23.794 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:25:23.794 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:23.797 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.814+0000 7ff456b32640 1 Processor -- start 2024-03-29T11:25:23.797 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.814+0000 7ff456b32640 1 -- start start 2024-03-29T11:25:23.797 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.814+0000 7ff456b32640 10 monclient: init 2024-03-29T11:25:23.799 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.815+0000 7ff456b32640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:23.799 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.815+0000 7ff456b32640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:23.799 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.815+0000 7ff456b32640 10 monclient(hunting): picked mon.noname-a con 0x7ff4501457f0 addr v1:172.21.15.135:6789/0 2024-03-29T11:25:23.799 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.815+0000 7ff456b32640 10 monclient(hunting): picked mon.noname-c con 0x7ff45013f650 addr v1:172.21.15.194:6789/0 2024-03-29T11:25:23.799 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.815+0000 7ff456b32640 10 monclient(hunting): picked mon.noname-b con 0x7ff4500a9d70 addr v1:172.21.15.135:6790/0 2024-03-29T11:25:23.799 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.815+0000 7ff456b32640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4500aaef0 con 0x7ff4501457f0 2024-03-29T11:25:23.799 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.815+0000 7ff456b32640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff450145bd0 con 0x7ff45013f650 2024-03-29T11:25:23.799 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.815+0000 7ff456b32640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff450145eb0 con 0x7ff4500a9d70 2024-03-29T11:25:23.799 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.815+0000 7ff456b32640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:23.799 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.815+0000 7ff456b32640 10 monclient(hunting): authenticate will time out at 831.183823s 2024-03-29T11:25:23.802 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.819+0000 7ff4548ce640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7ff4501457f0 0x7ff45013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:53581/0 (socket says 172.21.15.135:53581) 2024-03-29T11:25:23.803 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.819+0000 7ff4548ce640 1 -- 172.21.15.135:0/2092478652 learned_addr learned my addr 172.21.15.135:0/2092478652 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:25:23.807 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.823+0000 7ff44f7fe640 1 -- 172.21.15.135:0/2092478652 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763651301 0 0) 0x7ff4500aaef0 con 0x7ff4501457f0 2024-03-29T11:25:23.807 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.823+0000 7ff44f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:23.807 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.823+0000 7ff44f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:23.807 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.823+0000 7ff44f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:23.807 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.823+0000 7ff44f7fe640 10 monclient(hunting): my global_id is 4269 2024-03-29T11:25:23.807 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.823+0000 7ff44f7fe640 1 -- 172.21.15.135:0/2092478652 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff438003520 con 0x7ff4501457f0 2024-03-29T11:25:23.807 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.823+0000 7ff44f7fe640 1 -- 172.21.15.135:0/2092478652 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1875641806 0 0) 0x7ff450145eb0 con 0x7ff4500a9d70 2024-03-29T11:25:23.807 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.823+0000 7ff44f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:23.807 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.823+0000 7ff44f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:23.807 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.823+0000 7ff44f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:23.807 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.823+0000 7ff44f7fe640 10 monclient(hunting): my global_id is 4250 2024-03-29T11:25:23.807 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.823+0000 7ff44f7fe640 1 -- 172.21.15.135:0/2092478652 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4500aaef0 con 0x7ff4500a9d70 2024-03-29T11:25:23.808 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.824+0000 7ff44f7fe640 1 -- 172.21.15.135:0/2092478652 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 149745452 0 0) 0x7ff4500aaef0 con 0x7ff4500a9d70 2024-03-29T11:25:23.808 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.824+0000 7ff44f7fe640 1 -- 172.21.15.135:0/2092478652 >> v1:172.21.15.194:6789/0 conn(0x7ff45013f650 legacy=0x7ff4500b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:23.808 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.824+0000 7ff44f7fe640 1 -- 172.21.15.135:0/2092478652 >> v1:172.21.15.135:6789/0 conn(0x7ff4501457f0 legacy=0x7ff45013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:23.808 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.824+0000 7ff44f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:23.808 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.824+0000 7ff44f7fe640 1 monclient: found mon.noname-b 2024-03-29T11:25:23.808 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.824+0000 7ff44f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:23.809 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.824+0000 7ff44f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:25:23.809 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.824+0000 7ff44f7fe640 1 -- 172.21.15.135:0/2092478652 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff450146340 con 0x7ff4500a9d70 2024-03-29T11:25:23.809 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.824+0000 7ff44f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:23.809 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.824+0000 7ff44f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:23.809 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.824+0000 7ff44f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:23.809 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.824+0000 7ff456b32640 5 monclient: authenticate success, global_id 4250 2024-03-29T11:25:23.809 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.824+0000 7ff456b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:23.809 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.824+0000 7ff44f7fe640 1 -- 172.21.15.135:0/2092478652 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7ff4480015f0 con 0x7ff4500a9d70 2024-03-29T11:25:23.809 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.824+0000 7ff44f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:23.809 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.824+0000 7ff44f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:23.809 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.824+0000 7ff44f7fe640 10 monclient: dump: 2024-03-29T11:25:23.809 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:23.809 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:23.809 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:23.809 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:23.810 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:23.810 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:23.810 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:23.810 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:23.810 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:23.810 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:23.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.824+0000 7ff456b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:23.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.824+0000 7ff44f7fe640 1 -- 172.21.15.135:0/2092478652 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4480017d0 con 0x7ff4500a9d70 2024-03-29T11:25:23.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.824+0000 7ff44f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:23.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.824+0000 7ff456b32640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:23.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.824+0000 7ff456b32640 4 set_mon_vals no callback set 2024-03-29T11:25:23.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.824+0000 7ff44f7fe640 1 -- 172.21.15.135:0/2092478652 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7ff4480015f0 con 0x7ff4500a9d70 2024-03-29T11:25:23.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.824+0000 7ff44f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:23.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.824+0000 7ff44f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:23.811 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.824+0000 7ff44f7fe640 10 monclient: dump: 2024-03-29T11:25:23.811 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:23.811 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:23.811 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:23.811 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:23.811 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:23.811 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:23.811 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:23.811 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:23.811 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:23.811 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:23.813 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.825+0000 7ff456b32640 10 monclient: shutdown 2024-03-29T11:25:23.813 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.825+0000 7ff456b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:23.813 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.825+0000 7ff456b32640 1 -- 172.21.15.135:0/2092478652 >> v1:172.21.15.135:6790/0 conn(0x7ff4500a9d70 legacy=0x7ff45014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:23.813 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.826+0000 7ff456b32640 1 -- 172.21.15.135:0/2092478652 shutdown_connections 2024-03-29T11:25:23.813 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.826+0000 7ff456b32640 1 -- 172.21.15.135:0/2092478652 >> 172.21.15.135:0/2092478652 conn(0x7ff450066a00 msgr2=0x7ff450066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:23.813 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.826+0000 7ff456b32640 1 -- 172.21.15.135:0/2092478652 shutdown_connections 2024-03-29T11:25:23.813 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.826+0000 7ff456b32640 1 -- 172.21.15.135:0/2092478652 wait complete. 2024-03-29T11:25:23.813 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.827+0000 7ff456b32640 10 monclient: build_initial_monmap 2024-03-29T11:25:23.813 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.827+0000 7ff456b32640 10 monclient: monmap: 2024-03-29T11:25:23.814 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:25:23.814 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:23.814 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:25:23.814 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:25:23.814 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:23.814 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:23.814 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:25:23.814 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:25:23.814 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:25:23.814 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:23.814 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.827+0000 7ff456b32640 1 librados: starting msgr at 2024-03-29T11:25:23.814 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.827+0000 7ff456b32640 1 librados: starting objecter 2024-03-29T11:25:23.814 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.827+0000 7ff456b32640 1 Processor -- start 2024-03-29T11:25:23.814 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.828+0000 7ff456b32640 1 -- start start 2024-03-29T11:25:23.819 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.828+0000 7ff456b32640 1 librados: setting wanted keys 2024-03-29T11:25:23.819 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.828+0000 7ff456b32640 1 librados: calling monclient init 2024-03-29T11:25:23.820 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.828+0000 7ff456b32640 10 monclient: init 2024-03-29T11:25:23.820 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.829+0000 7ff456b32640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:23.820 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.829+0000 7ff456b32640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:23.820 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.829+0000 7ff456b32640 10 monclient(hunting): picked mon.noname-b con 0x7ff4500a9d70 addr v1:172.21.15.194:6789/0 2024-03-29T11:25:23.820 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.829+0000 7ff456b32640 10 monclient(hunting): picked mon.noname-c con 0x7ff45013f650 addr v1:172.21.15.135:6790/0 2024-03-29T11:25:23.820 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.829+0000 7ff456b32640 10 monclient(hunting): picked mon.noname-a con 0x7ff4501457f0 addr v1:172.21.15.135:6789/0 2024-03-29T11:25:23.820 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.829+0000 7ff456b32640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4501e73a0 con 0x7ff4501457f0 2024-03-29T11:25:23.820 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.829+0000 7ff456b32640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4501e7f90 con 0x7ff4500a9d70 2024-03-29T11:25:23.820 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.829+0000 7ff456b32640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4501e8120 con 0x7ff45013f650 2024-03-29T11:25:23.820 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.829+0000 7ff456b32640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:23.820 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.829+0000 7ff456b32640 10 monclient(hunting): authenticate will time out at 831.197417s 2024-03-29T11:25:23.820 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.833+0000 7ff4550cf640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7ff4501457f0 0x7ff4501ee7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:53631/0 (socket says 172.21.15.135:53631) 2024-03-29T11:25:23.820 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.833+0000 7ff4550cf640 1 -- 172.21.15.135:0/3643440034 learned_addr learned my addr 172.21.15.135:0/3643440034 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:25:23.820 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.833+0000 7ff44d7fa640 10 client.?.objecter ms_handle_connect 0x7ff4501457f0 2024-03-29T11:25:23.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.833+0000 7ff44d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:23.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.833+0000 7ff44d7fa640 1 -- 172.21.15.135:0/3643440034 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3114795156 0 0) 0x7ff4501e73a0 con 0x7ff4501457f0 2024-03-29T11:25:23.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.833+0000 7ff44d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:23.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.833+0000 7ff44d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:23.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.834+0000 7ff44d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:23.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.834+0000 7ff44d7fa640 10 monclient(hunting): my global_id is 4272 2024-03-29T11:25:23.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.834+0000 7ff44d7fa640 1 -- 172.21.15.135:0/3643440034 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff43c004eb0 con 0x7ff4501457f0 2024-03-29T11:25:23.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.834+0000 7ff44d7fa640 1 -- 172.21.15.135:0/3643440034 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3732807571 0 0) 0x7ff43c004eb0 con 0x7ff4501457f0 2024-03-29T11:25:23.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.834+0000 7ff44d7fa640 1 -- 172.21.15.135:0/3643440034 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4501e73a0 con 0x7ff4501457f0 2024-03-29T11:25:23.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.834+0000 7ff44d7fa640 10 client.?.objecter ms_handle_connect 0x7ff45013f650 2024-03-29T11:25:23.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.834+0000 7ff44d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:23.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.834+0000 7ff44d7fa640 1 -- 172.21.15.135:0/3643440034 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7ff448001670 con 0x7ff4501457f0 2024-03-29T11:25:23.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.834+0000 7ff44d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:23.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.834+0000 7ff44d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:23.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.834+0000 7ff44d7fa640 10 monclient(hunting): dump: 2024-03-29T11:25:23.822 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:23.822 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:23.822 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:23.822 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:23.822 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:23.822 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:23.822 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:23.822 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:23.822 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:23.822 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:23.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.834+0000 7ff44d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:23.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.834+0000 7ff44d7fa640 1 -- 172.21.15.135:0/3643440034 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 88926301 0 0) 0x7ff4501e8120 con 0x7ff45013f650 2024-03-29T11:25:23.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.834+0000 7ff44d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:23.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.834+0000 7ff44d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:23.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.834+0000 7ff44d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:23.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.834+0000 7ff44d7fa640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:25:23.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.834+0000 7ff44d7fa640 1 -- 172.21.15.135:0/3643440034 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff43c004eb0 con 0x7ff45013f650 2024-03-29T11:25:23.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.835+0000 7ff44d7fa640 1 -- 172.21.15.135:0/3643440034 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3063285200 0 0) 0x7ff4501e73a0 con 0x7ff4501457f0 2024-03-29T11:25:23.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.835+0000 7ff44d7fa640 1 -- 172.21.15.135:0/3643440034 >> v1:172.21.15.135:6790/0 conn(0x7ff45013f650 legacy=0x7ff45006e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:23.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.835+0000 7ff44d7fa640 1 -- 172.21.15.135:0/3643440034 >> v1:172.21.15.194:6789/0 conn(0x7ff4500a9d70 legacy=0x7ff4501e7880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:23.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.836+0000 7ff44d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:25:23.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.838+0000 7ff44d7fa640 1 monclient: found mon.a 2024-03-29T11:25:23.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.838+0000 7ff44d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:23.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.838+0000 7ff44d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:25:23.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.838+0000 7ff44d7fa640 1 -- 172.21.15.135:0/3643440034 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45006e810 con 0x7ff4501457f0 2024-03-29T11:25:23.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.838+0000 7ff44d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:25:23.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.838+0000 7ff44d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:25:23.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.838+0000 7ff44d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:23.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.838+0000 7ff456b32640 5 monclient: authenticate success, global_id 4272 2024-03-29T11:25:23.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.838+0000 7ff456b32640 10 monclient: _renew_subs 2024-03-29T11:25:23.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.838+0000 7ff456b32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:25:23.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.838+0000 7ff456b32640 1 -- 172.21.15.135:0/3643440034 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff45013fda0 con 0x7ff4501457f0 2024-03-29T11:25:23.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.838+0000 7ff456b32640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:23.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.838+0000 7ff456b32640 10 monclient: _renew_subs 2024-03-29T11:25:23.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.838+0000 7ff456b32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:25:23.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.838+0000 7ff456b32640 1 -- 172.21.15.135:0/3643440034 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4501426a0 con 0x7ff4501457f0 2024-03-29T11:25:23.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.838+0000 7ff456b32640 1 librados: init done 2024-03-29T11:25:23.825 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.838+0000 7ff44d7fa640 1 -- 172.21.15.135:0/3643440034 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4480035f0 con 0x7ff4501457f0 2024-03-29T11:25:23.825 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.838+0000 7ff44d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:23.825 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.838+0000 7ff44d7fa640 1 -- 172.21.15.135:0/3643440034 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7ff448003c40 con 0x7ff4501457f0 2024-03-29T11:25:23.825 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.838+0000 7ff44d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:23.825 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.838+0000 7ff44d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:23.825 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.838+0000 7ff44d7fa640 10 monclient: dump: 2024-03-29T11:25:23.825 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:23.825 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:23.825 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:23.825 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:23.825 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:23.825 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:23.825 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:23.826 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:23.826 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:23.826 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:23.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.841+0000 7ff456b32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:23.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.841+0000 7ff456b32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:23.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.841+0000 7ff456b32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:25:23.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.841+0000 7ff456b32640 1 -- 172.21.15.135:0/3643440034 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff45006fa70 con 0x7ff4501457f0 2024-03-29T11:25:23.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.842+0000 7ff44effd640 4 set_mon_vals no callback set 2024-03-29T11:25:23.830 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.847+0000 7ff44d7fa640 1 -- 172.21.15.135:0/3643440034 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7ff448003da0 con 0x7ff4501457f0 2024-03-29T11:25:23.831 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.847+0000 7ff44d7fa640 1 -- 172.21.15.135:0/3643440034 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7ff448084c40 con 0x7ff4501457f0 2024-03-29T11:25:23.831 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.848+0000 7ff44d7fa640 10 client.4272.objecter ms_dispatch 0x7ff450000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:23.831 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.848+0000 7ff44d7fa640 3 client.4272.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:23.831 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.848+0000 7ff44d7fa640 3 client.4272.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:23.831 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.848+0000 7ff44d7fa640 20 client.4272.objecter dump_active .. 0 homeless 2024-03-29T11:25:23.832 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.849+0000 7ff4548ce640 10 monclient: get_auth_request con 0x7ff43c06daa0 auth_method 0 2024-03-29T11:25:23.835 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.850+0000 7ff44d7fa640 10 client.4272.objecter ms_handle_connect 0x7ff43c06daa0 2024-03-29T11:25:23.835 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.850+0000 7ff44d7fa640 1 -- 172.21.15.135:0/3643440034 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff448050350 con 0x7ff4501457f0 2024-03-29T11:25:23.835 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.850+0000 7ff44d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:23.835 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:23.850+0000 7ff44d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:24.154 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:24.168+0000 7ff456b32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:24.154 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:24.168+0000 7ff456b32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:24.154 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:24.168+0000 7ff456b32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:25:24.154 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:24.168+0000 7ff456b32640 1 -- 172.21.15.135:0/3643440034 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff450075f10 con 0x7ff4501457f0 2024-03-29T11:25:24.155 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:24.171+0000 7ff44d7fa640 1 -- 172.21.15.135:0/3643440034 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7ff448052450 con 0x7ff4501457f0 2024-03-29T11:25:24.155 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:24.171+0000 7ff44d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:24.155 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:24.171+0000 7ff44d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:24.155 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:25:24.155 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:24.158 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:24.175+0000 7ff456b32640 10 librados: watch_flush enter 2024-03-29T11:25:24.160 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:24.177+0000 7ff456b32640 10 librados: watch_flush exit 2024-03-29T11:25:24.161 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:24.177+0000 7ff456b32640 20 client.4272.objecter shutdown clearing up homeless session... 2024-03-29T11:25:24.161 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:24.177+0000 7ff456b32640 10 client.4272.objecter successfully canceled tick 2024-03-29T11:25:24.161 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:24.178+0000 7ff456b32640 1 -- 172.21.15.135:0/3643440034 >> v1:172.21.15.135:6816/3469875285 conn(0x7ff43c06daa0 legacy=0x7ff43c06ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:24.161 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:24.178+0000 7ff456b32640 10 monclient: shutdown 2024-03-29T11:25:24.161 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:24.178+0000 7ff456b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:24.161 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:24.178+0000 7ff456b32640 1 -- 172.21.15.135:0/3643440034 >> v1:172.21.15.135:6789/0 conn(0x7ff4501457f0 legacy=0x7ff4501ee7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:24.162 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:24.178+0000 7ff456b32640 1 -- 172.21.15.135:0/3643440034 shutdown_connections 2024-03-29T11:25:24.162 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:24.178+0000 7ff456b32640 1 -- 172.21.15.135:0/3643440034 >> 172.21.15.135:0/3643440034 conn(0x7ff450066a00 msgr2=0x7ff45013b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:24.162 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:24.179+0000 7ff456b32640 1 -- 172.21.15.135:0/3643440034 shutdown_connections 2024-03-29T11:25:24.162 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:24.179+0000 7ff456b32640 1 -- 172.21.15.135:0/3643440034 wait complete. 2024-03-29T11:25:24.162 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:24.179+0000 7ff456b32640 1 librados: shutdown 2024-03-29T11:25:24.181 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:30.988 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:31.136 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.152+0000 7f2c03c74640 10 monclient: get_monmap_and_config 2024-03-29T11:25:31.136 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.152+0000 7f2c03c74640 10 monclient: build_initial_monmap 2024-03-29T11:25:31.136 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.152+0000 7f2c03c74640 10 monclient: monmap: 2024-03-29T11:25:31.136 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:25:31.136 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:31.136 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:25:31.153122+0000 2024-03-29T11:25:31.136 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:25:31.153122+0000 2024-03-29T11:25:31.137 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:31.137 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:31.137 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:25:31.137 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:25:31.137 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:25:31.137 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:31.142 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.153+0000 7f2c03c74640 1 Processor -- start 2024-03-29T11:25:31.142 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.153+0000 7f2c03c74640 1 -- start start 2024-03-29T11:25:31.142 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.153+0000 7f2c03c74640 10 monclient: init 2024-03-29T11:25:31.142 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.154+0000 7f2c03c74640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:31.142 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.154+0000 7f2c03c74640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:31.142 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.154+0000 7f2c03c74640 10 monclient(hunting): picked mon.noname-a con 0x7f2bfc1422b0 addr v1:172.21.15.135:6789/0 2024-03-29T11:25:31.142 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.154+0000 7f2c03c74640 10 monclient(hunting): picked mon.noname-b con 0x7f2bfc142690 addr v1:172.21.15.135:6790/0 2024-03-29T11:25:31.143 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.154+0000 7f2c03c74640 10 monclient(hunting): picked mon.noname-c con 0x7f2bfc0b17d0 addr v1:172.21.15.194:6789/0 2024-03-29T11:25:31.143 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.154+0000 7f2c03c74640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bfc142c80 con 0x7f2bfc1422b0 2024-03-29T11:25:31.143 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.154+0000 7f2c03c74640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bfc0aae40 con 0x7f2bfc0b17d0 2024-03-29T11:25:31.143 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.154+0000 7f2c03c74640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bfc13fbd0 con 0x7f2bfc142690 2024-03-29T11:25:31.143 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.154+0000 7f2c03c74640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:31.143 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.154+0000 7f2c03c74640 10 monclient(hunting): authenticate will time out at 838.522946s 2024-03-29T11:25:31.143 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.154+0000 7f2c02211640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f2bfc0b17d0 0x7f2bfc0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:39440/0 (socket says 172.21.15.135:39440) 2024-03-29T11:25:31.143 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.154+0000 7f2c02211640 1 -- 172.21.15.135:0/126030584 learned_addr learned my addr 172.21.15.135:0/126030584 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:25:31.143 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.155+0000 7f2c00a0e640 1 -- 172.21.15.135:0/126030584 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2605661292 0 0) 0x7f2bfc0aae40 con 0x7f2bfc0b17d0 2024-03-29T11:25:31.143 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.155+0000 7f2c00a0e640 10 monclient(hunting): _negotiate 2024-03-29T11:25:31.143 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.155+0000 7f2c00a0e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:31.143 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.155+0000 7f2c00a0e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:31.143 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.155+0000 7f2c00a0e640 10 monclient(hunting): my global_id is 4267 2024-03-29T11:25:31.143 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.155+0000 7f2c00a0e640 1 -- 172.21.15.135:0/126030584 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2be8003520 con 0x7f2bfc0b17d0 2024-03-29T11:25:31.144 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c00a0e640 1 -- 172.21.15.135:0/126030584 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3008881900 0 0) 0x7f2bfc142c80 con 0x7f2bfc1422b0 2024-03-29T11:25:31.144 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c00a0e640 10 monclient(hunting): _negotiate 2024-03-29T11:25:31.144 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c00a0e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:31.144 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c00a0e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:31.144 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c00a0e640 10 monclient(hunting): my global_id is 4275 2024-03-29T11:25:31.144 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c00a0e640 1 -- 172.21.15.135:0/126030584 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bfc0aae40 con 0x7f2bfc1422b0 2024-03-29T11:25:31.144 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c00a0e640 1 -- 172.21.15.135:0/126030584 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2808246593 0 0) 0x7f2be8003520 con 0x7f2bfc0b17d0 2024-03-29T11:25:31.144 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c00a0e640 1 -- 172.21.15.135:0/126030584 >> v1:172.21.15.135:6790/0 conn(0x7f2bfc142690 legacy=0x7f2bfc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:31.144 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c00a0e640 1 -- 172.21.15.135:0/126030584 >> v1:172.21.15.135:6789/0 conn(0x7f2bfc1422b0 legacy=0x7f2bfc13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:31.144 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c00a0e640 10 monclient: _finish_hunting 0 2024-03-29T11:25:31.144 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c00a0e640 1 monclient: found mon.noname-c 2024-03-29T11:25:31.144 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c00a0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:31.145 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c00a0e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:25:31.145 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c00a0e640 1 -- 172.21.15.135:0/126030584 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bfc151940 con 0x7f2bfc0b17d0 2024-03-29T11:25:31.145 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c00a0e640 10 monclient: _finish_auth 0 2024-03-29T11:25:31.145 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c00a0e640 10 monclient: _check_auth_tickets 2024-03-29T11:25:31.145 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c00a0e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:31.145 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c03c74640 5 monclient: authenticate success, global_id 4267 2024-03-29T11:25:31.145 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c03c74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:31.145 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c00a0e640 1 -- 172.21.15.135:0/126030584 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f2bf4001a10 con 0x7f2bfc0b17d0 2024-03-29T11:25:31.145 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c00a0e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:31.145 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c00a0e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:31.145 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c00a0e640 10 monclient: dump: 2024-03-29T11:25:31.145 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:31.145 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:31.145 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:31.146 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:31.146 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:31.146 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:31.146 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:31.146 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:31.146 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:31.146 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:31.146 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c03c74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:31.146 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c00a0e640 1 -- 172.21.15.135:0/126030584 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bf4001c30 con 0x7f2bfc0b17d0 2024-03-29T11:25:31.146 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c00a0e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:31.146 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c03c74640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:31.146 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.156+0000 7f2c03c74640 4 set_mon_vals no callback set 2024-03-29T11:25:31.146 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.157+0000 7f2c00a0e640 1 -- 172.21.15.135:0/126030584 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f2bf4001f40 con 0x7f2bfc0b17d0 2024-03-29T11:25:31.146 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.157+0000 7f2c00a0e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:31.147 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.157+0000 7f2c00a0e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:31.147 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.157+0000 7f2c00a0e640 10 monclient: dump: 2024-03-29T11:25:31.147 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:31.147 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:31.147 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:31.147 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:31.147 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:31.147 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:31.147 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:31.147 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:31.147 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:31.147 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:31.147 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.157+0000 7f2c03c74640 10 monclient: shutdown 2024-03-29T11:25:31.147 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.157+0000 7f2c03c74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:31.147 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.157+0000 7f2c03c74640 1 -- 172.21.15.135:0/126030584 >> v1:172.21.15.194:6789/0 conn(0x7f2bfc0b17d0 legacy=0x7f2bfc0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:31.148 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.158+0000 7f2c03c74640 1 -- 172.21.15.135:0/126030584 shutdown_connections 2024-03-29T11:25:31.148 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.158+0000 7f2c03c74640 1 -- 172.21.15.135:0/126030584 >> 172.21.15.135:0/126030584 conn(0x7f2bfc066a00 msgr2=0x7f2bfc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:31.148 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.158+0000 7f2c03c74640 1 -- 172.21.15.135:0/126030584 shutdown_connections 2024-03-29T11:25:31.148 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.158+0000 7f2c03c74640 1 -- 172.21.15.135:0/126030584 wait complete. 2024-03-29T11:25:31.152 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.159+0000 7f2c03c74640 10 monclient: build_initial_monmap 2024-03-29T11:25:31.153 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.159+0000 7f2c03c74640 10 monclient: monmap: 2024-03-29T11:25:31.153 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:25:31.153 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:31.153 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:25:31.153 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:25:31.153 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:31.153 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:31.154 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:25:31.154 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:25:31.154 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:25:31.154 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:31.154 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.159+0000 7f2c03c74640 1 librados: starting msgr at 2024-03-29T11:25:31.154 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.159+0000 7f2c03c74640 1 librados: starting objecter 2024-03-29T11:25:31.154 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.159+0000 7f2c03c74640 1 Processor -- start 2024-03-29T11:25:31.154 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.160+0000 7f2c03c74640 1 -- start start 2024-03-29T11:25:31.154 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.160+0000 7f2c03c74640 1 librados: setting wanted keys 2024-03-29T11:25:31.154 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.160+0000 7f2c03c74640 1 librados: calling monclient init 2024-03-29T11:25:31.154 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.160+0000 7f2c03c74640 10 monclient: init 2024-03-29T11:25:31.154 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.161+0000 7f2c03c74640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:31.155 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.161+0000 7f2c03c74640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:31.155 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.161+0000 7f2c03c74640 10 monclient(hunting): picked mon.noname-b con 0x7f2bfc0b17d0 addr v1:172.21.15.194:6789/0 2024-03-29T11:25:31.155 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.161+0000 7f2c03c74640 10 monclient(hunting): picked mon.noname-c con 0x7f2bfc1422b0 addr v1:172.21.15.135:6790/0 2024-03-29T11:25:31.155 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.161+0000 7f2c03c74640 10 monclient(hunting): picked mon.noname-a con 0x7f2bfc142690 addr v1:172.21.15.135:6789/0 2024-03-29T11:25:31.155 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.161+0000 7f2c03c74640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bfc1328f0 con 0x7f2bfc142690 2024-03-29T11:25:31.155 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.161+0000 7f2c01a10640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f2bfc0b17d0 0x7f2bfc1321e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:40146/0 (socket says 172.21.15.135:40146) 2024-03-29T11:25:31.155 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.161+0000 7f2c01a10640 1 -- 172.21.15.135:0/2639902306 learned_addr learned my addr 172.21.15.135:0/2639902306 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:25:31.155 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.161+0000 7f2c03c74640 1 -- 172.21.15.135:0/2639902306 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bfc06eab0 con 0x7f2bfc0b17d0 2024-03-29T11:25:31.155 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.161+0000 7f2c03c74640 1 -- 172.21.15.135:0/2639902306 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bfc06ed90 con 0x7f2bfc1422b0 2024-03-29T11:25:31.155 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.161+0000 7f2c03c74640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:31.156 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.161+0000 7f2c03c74640 10 monclient(hunting): authenticate will time out at 838.529604s 2024-03-29T11:25:31.156 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.161+0000 7f2bf27fc640 10 client.?.objecter ms_handle_connect 0x7f2bfc0b17d0 2024-03-29T11:25:31.156 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.161+0000 7f2bf27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:31.156 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.161+0000 7f2bf27fc640 10 client.?.objecter ms_handle_connect 0x7f2bfc1422b0 2024-03-29T11:25:31.156 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.161+0000 7f2bf27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:31.156 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.161+0000 7f2bf27fc640 1 -- 172.21.15.135:0/2639902306 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 325264489 0 0) 0x7f2bfc06eab0 con 0x7f2bfc0b17d0 2024-03-29T11:25:31.156 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.161+0000 7f2bf27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:31.156 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.161+0000 7f2bf27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:31.156 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.161+0000 7f2bf27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:31.156 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.161+0000 7f2bf27fc640 10 monclient(hunting): my global_id is 4270 2024-03-29T11:25:31.156 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.161+0000 7f2bf27fc640 1 -- 172.21.15.135:0/2639902306 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2be0004eb0 con 0x7f2bfc0b17d0 2024-03-29T11:25:31.156 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.162+0000 7f2bf27fc640 1 -- 172.21.15.135:0/2639902306 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572576263 0 0) 0x7f2bfc06ed90 con 0x7f2bfc1422b0 2024-03-29T11:25:31.157 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.162+0000 7f2bf27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:31.157 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.162+0000 7f2bf27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:31.157 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.162+0000 7f2bf27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:31.157 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.162+0000 7f2bf27fc640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:25:31.157 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.162+0000 7f2bf27fc640 1 -- 172.21.15.135:0/2639902306 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bfc06eab0 con 0x7f2bfc1422b0 2024-03-29T11:25:31.157 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.162+0000 7f2bf27fc640 10 client.?.objecter ms_handle_connect 0x7f2bfc142690 2024-03-29T11:25:31.157 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.162+0000 7f2bf27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:31.157 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.162+0000 7f2bf27fc640 1 -- 172.21.15.135:0/2639902306 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 334455743 0 0) 0x7f2be0004eb0 con 0x7f2bfc0b17d0 2024-03-29T11:25:31.157 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.162+0000 7f2bf27fc640 1 -- 172.21.15.135:0/2639902306 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2bfc06ed90 con 0x7f2bfc0b17d0 2024-03-29T11:25:31.157 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.162+0000 7f2bf27fc640 1 -- 172.21.15.135:0/2639902306 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488380481 0 0) 0x7f2bfc1328f0 con 0x7f2bfc142690 2024-03-29T11:25:31.157 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.162+0000 7f2bf27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:31.158 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.162+0000 7f2bf27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:31.158 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.162+0000 7f2bf27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:31.158 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.162+0000 7f2bf27fc640 10 monclient(hunting): my global_id is 4278 2024-03-29T11:25:31.158 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.162+0000 7f2bf27fc640 1 -- 172.21.15.135:0/2639902306 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2be0004eb0 con 0x7f2bfc142690 2024-03-29T11:25:31.158 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.162+0000 7f2bf27fc640 1 -- 172.21.15.135:0/2639902306 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f2bf80019c0 con 0x7f2bfc0b17d0 2024-03-29T11:25:31.158 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.162+0000 7f2bf27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:31.158 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.162+0000 7f2bf27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:31.158 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.162+0000 7f2bf27fc640 10 monclient(hunting): dump: 2024-03-29T11:25:31.158 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:31.158 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:31.158 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:31.159 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:31.159 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:31.159 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:31.159 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:31.159 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:31.159 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:31.159 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:31.159 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.162+0000 7f2bf27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:31.159 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.162+0000 7f2bf27fc640 1 -- 172.21.15.135:0/2639902306 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4041885278 0 0) 0x7f2bfc06ed90 con 0x7f2bfc0b17d0 2024-03-29T11:25:31.159 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.162+0000 7f2bf27fc640 1 -- 172.21.15.135:0/2639902306 >> v1:172.21.15.135:6790/0 conn(0x7f2bfc1422b0 legacy=0x7f2bfc071570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:31.159 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.163+0000 7f2bf27fc640 1 -- 172.21.15.135:0/2639902306 >> v1:172.21.15.135:6789/0 conn(0x7f2bfc142690 legacy=0x7f2bfc1e6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:31.159 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.163+0000 7f2bf27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:25:31.160 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.163+0000 7f2bf27fc640 1 monclient: found mon.b 2024-03-29T11:25:31.160 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.163+0000 7f2bf27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:31.160 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.163+0000 7f2bf27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:25:31.160 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.163+0000 7f2bf27fc640 1 -- 172.21.15.135:0/2639902306 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bfc131ca0 con 0x7f2bfc0b17d0 2024-03-29T11:25:31.160 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.163+0000 7f2bf27fc640 10 monclient: _finish_auth 0 2024-03-29T11:25:31.160 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.163+0000 7f2bf27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:25:31.160 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.163+0000 7f2bf27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:31.160 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.163+0000 7f2c03c74640 5 monclient: authenticate success, global_id 4270 2024-03-29T11:25:31.160 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.163+0000 7f2c03c74640 10 monclient: _renew_subs 2024-03-29T11:25:31.160 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.163+0000 7f2c03c74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:25:31.160 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.163+0000 7f2c03c74640 1 -- 172.21.15.135:0/2639902306 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bfc13ba60 con 0x7f2bfc0b17d0 2024-03-29T11:25:31.161 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.163+0000 7f2c03c74640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:31.161 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.163+0000 7f2c03c74640 10 monclient: _renew_subs 2024-03-29T11:25:31.161 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.163+0000 7f2c03c74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:25:31.161 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.163+0000 7f2c03c74640 1 -- 172.21.15.135:0/2639902306 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bfc1deb50 con 0x7f2bfc0b17d0 2024-03-29T11:25:31.161 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.163+0000 7f2c03c74640 1 librados: init done 2024-03-29T11:25:31.161 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.163+0000 7f2bf27fc640 1 -- 172.21.15.135:0/2639902306 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bf8001b20 con 0x7f2bfc0b17d0 2024-03-29T11:25:31.161 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.163+0000 7f2bf27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:31.161 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.163+0000 7f2bf27fc640 1 -- 172.21.15.135:0/2639902306 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f2bf8001d80 con 0x7f2bfc0b17d0 2024-03-29T11:25:31.161 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.163+0000 7f2bf27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:31.161 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.163+0000 7f2bf27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:31.161 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.163+0000 7f2bf27fc640 10 monclient: dump: 2024-03-29T11:25:31.162 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:31.162 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:31.162 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:31.162 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:31.162 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:31.162 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:31.162 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:31.162 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:31.163 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:31.163 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:31.163 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.163+0000 7f2bf3fff640 4 set_mon_vals no callback set 2024-03-29T11:25:31.163 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.165+0000 7f2c03c74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:31.163 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.165+0000 7f2c03c74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:31.163 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.165+0000 7f2bf27fc640 1 -- 172.21.15.135:0/2639902306 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f2bf8003b40 con 0x7f2bfc0b17d0 2024-03-29T11:25:31.163 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.165+0000 7f2c03c74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:25:31.163 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.165+0000 7f2c03c74640 1 -- 172.21.15.135:0/2639902306 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be4005080 con 0x7f2bfc0b17d0 2024-03-29T11:25:31.163 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.166+0000 7f2bf27fc640 1 -- 172.21.15.135:0/2639902306 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f2bf8084b10 con 0x7f2bfc0b17d0 2024-03-29T11:25:31.163 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.166+0000 7f2bf27fc640 10 client.4270.objecter ms_dispatch 0x7f2bfc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:31.164 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.166+0000 7f2bf27fc640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:31.164 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.166+0000 7f2bf27fc640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:31.164 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.166+0000 7f2bf27fc640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-29T11:25:31.164 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.167+0000 7f2c0120f640 10 monclient: get_auth_request con 0x7f2be006dcb0 auth_method 0 2024-03-29T11:25:31.164 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.167+0000 7f2bf27fc640 10 client.4270.objecter ms_handle_connect 0x7f2be006dcb0 2024-03-29T11:25:31.165 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.173+0000 7f2bf27fc640 1 -- 172.21.15.135:0/2639902306 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2bf80503f0 con 0x7f2bfc0b17d0 2024-03-29T11:25:31.165 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.173+0000 7f2bf27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:31.165 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.173+0000 7f2bf27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:31.546 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.562+0000 7f2c03c74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:31.546 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.562+0000 7f2c03c74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:31.546 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.562+0000 7f2c03c74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:25:31.546 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.562+0000 7f2c03c74640 1 -- 172.21.15.135:0/2639902306 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2be4005330 con 0x7f2bfc0b17d0 2024-03-29T11:25:31.547 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.563+0000 7f2bf27fc640 1 -- 172.21.15.135:0/2639902306 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f2bf8052320 con 0x7f2bfc0b17d0 2024-03-29T11:25:31.547 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.563+0000 7f2bf27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:31.547 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.563+0000 7f2bf27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:31.547 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:25:31.548 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:31.552 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.568+0000 7f2c03c74640 10 librados: watch_flush enter 2024-03-29T11:25:31.552 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.568+0000 7f2c03c74640 10 librados: watch_flush exit 2024-03-29T11:25:31.552 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.568+0000 7f2c03c74640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-29T11:25:31.552 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.568+0000 7f2c03c74640 10 client.4270.objecter successfully canceled tick 2024-03-29T11:25:31.552 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.568+0000 7f2c03c74640 1 -- 172.21.15.135:0/2639902306 >> v1:172.21.15.135:6816/3469875285 conn(0x7f2be006dcb0 legacy=0x7f2be0070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:31.552 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.568+0000 7f2c03c74640 10 monclient: shutdown 2024-03-29T11:25:31.552 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.568+0000 7f2c03c74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:31.552 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.568+0000 7f2c03c74640 1 -- 172.21.15.135:0/2639902306 >> v1:172.21.15.194:6789/0 conn(0x7f2bfc0b17d0 legacy=0x7f2bfc1321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:31.552 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.568+0000 7f2c03c74640 1 -- 172.21.15.135:0/2639902306 shutdown_connections 2024-03-29T11:25:31.552 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.568+0000 7f2c03c74640 1 -- 172.21.15.135:0/2639902306 >> 172.21.15.135:0/2639902306 conn(0x7f2bfc066a00 msgr2=0x7f2bfc0b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:31.553 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.569+0000 7f2c03c74640 1 -- 172.21.15.135:0/2639902306 shutdown_connections 2024-03-29T11:25:31.553 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.569+0000 7f2c03c74640 1 -- 172.21.15.135:0/2639902306 wait complete. 2024-03-29T11:25:31.553 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:31.569+0000 7f2c03c74640 1 librados: shutdown 2024-03-29T11:25:31.571 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:38.378 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:38.528 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.544+0000 7fec58e7b640 10 monclient: get_monmap_and_config 2024-03-29T11:25:38.528 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.544+0000 7fec58e7b640 10 monclient: build_initial_monmap 2024-03-29T11:25:38.528 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.544+0000 7fec58e7b640 10 monclient: monmap: 2024-03-29T11:25:38.528 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:25:38.528 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:38.528 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:25:38.544834+0000 2024-03-29T11:25:38.528 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:25:38.544834+0000 2024-03-29T11:25:38.528 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:38.528 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:38.529 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:25:38.529 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:25:38.529 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:25:38.529 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:38.529 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.545+0000 7fec58e7b640 1 Processor -- start 2024-03-29T11:25:38.529 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.545+0000 7fec58e7b640 1 -- start start 2024-03-29T11:25:38.529 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.545+0000 7fec58e7b640 10 monclient: init 2024-03-29T11:25:38.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.546+0000 7fec58e7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:38.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.546+0000 7fec58e7b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:38.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.546+0000 7fec58e7b640 10 monclient(hunting): picked mon.noname-b con 0x7fec5406ade0 addr v1:172.21.15.135:6790/0 2024-03-29T11:25:38.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.546+0000 7fec58e7b640 10 monclient(hunting): picked mon.noname-c con 0x7fec540b06f0 addr v1:172.21.15.194:6789/0 2024-03-29T11:25:38.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.546+0000 7fec58e7b640 10 monclient(hunting): picked mon.noname-a con 0x7fec540721c0 addr v1:172.21.15.135:6789/0 2024-03-29T11:25:38.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.546+0000 7fec58e7b640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec5406fc90 con 0x7fec540721c0 2024-03-29T11:25:38.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.546+0000 7fec58e7b640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec5406ff70 con 0x7fec540b06f0 2024-03-29T11:25:38.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.546+0000 7fec58e7b640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec5406f070 con 0x7fec5406ade0 2024-03-29T11:25:38.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.546+0000 7fec58e7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:38.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.546+0000 7fec58e7b640 10 monclient(hunting): authenticate will time out at 845.914798s 2024-03-29T11:25:38.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.546+0000 7fec52d9d640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fec540721c0 0x7fec54151d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:33204/0 (socket says 172.21.15.135:33204) 2024-03-29T11:25:38.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.546+0000 7fec52d9d640 1 -- 172.21.15.135:0/2819522990 learned_addr learned my addr 172.21.15.135:0/2819522990 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:25:38.532 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.547+0000 7fec5159a640 1 -- 172.21.15.135:0/2819522990 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000196693 0 0) 0x7fec5406f070 con 0x7fec5406ade0 2024-03-29T11:25:38.532 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.547+0000 7fec5159a640 10 monclient(hunting): _negotiate 2024-03-29T11:25:38.532 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.547+0000 7fec5159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:38.532 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.547+0000 7fec5159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:38.532 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.547+0000 7fec5159a640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:25:38.532 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.547+0000 7fec5159a640 1 -- 172.21.15.135:0/2819522990 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec3c003520 con 0x7fec5406ade0 2024-03-29T11:25:38.532 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.547+0000 7fec5159a640 1 -- 172.21.15.135:0/2819522990 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813273564 0 0) 0x7fec5406fc90 con 0x7fec540721c0 2024-03-29T11:25:38.532 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.547+0000 7fec5159a640 10 monclient(hunting): _negotiate 2024-03-29T11:25:38.532 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.547+0000 7fec5159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:38.533 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.547+0000 7fec5159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:38.533 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.547+0000 7fec5159a640 10 monclient(hunting): my global_id is 4281 2024-03-29T11:25:38.533 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.547+0000 7fec5159a640 1 -- 172.21.15.135:0/2819522990 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec5406f070 con 0x7fec540721c0 2024-03-29T11:25:38.533 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.548+0000 7fec5159a640 1 -- 172.21.15.135:0/2819522990 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3146443949 0 0) 0x7fec3c003520 con 0x7fec5406ade0 2024-03-29T11:25:38.533 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.548+0000 7fec5159a640 1 -- 172.21.15.135:0/2819522990 >> v1:172.21.15.194:6789/0 conn(0x7fec540b06f0 legacy=0x7fec54071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:38.533 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.548+0000 7fec5159a640 1 -- 172.21.15.135:0/2819522990 >> v1:172.21.15.135:6789/0 conn(0x7fec540721c0 legacy=0x7fec54151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:38.533 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.548+0000 7fec5159a640 10 monclient: _finish_hunting 0 2024-03-29T11:25:38.533 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.548+0000 7fec5159a640 1 monclient: found mon.noname-b 2024-03-29T11:25:38.533 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.548+0000 7fec5159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:38.533 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.548+0000 7fec5159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:25:38.533 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.548+0000 7fec5159a640 1 -- 172.21.15.135:0/2819522990 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec540aeed0 con 0x7fec5406ade0 2024-03-29T11:25:38.533 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.548+0000 7fec5159a640 10 monclient: _finish_auth 0 2024-03-29T11:25:38.533 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.548+0000 7fec5159a640 10 monclient: _check_auth_tickets 2024-03-29T11:25:38.533 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.548+0000 7fec5159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:38.534 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.548+0000 7fec5159a640 1 -- 172.21.15.135:0/2819522990 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fec48001690 con 0x7fec5406ade0 2024-03-29T11:25:38.534 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.548+0000 7fec5159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:38.534 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.548+0000 7fec5159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:38.534 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.548+0000 7fec5159a640 10 monclient: dump: 2024-03-29T11:25:38.534 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:38.534 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:38.534 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:38.534 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:38.534 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:38.534 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:38.534 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:38.534 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:38.534 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:38.534 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:38.535 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.548+0000 7fec58e7b640 5 monclient: authenticate success, global_id 4262 2024-03-29T11:25:38.535 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.548+0000 7fec58e7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:38.535 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.549+0000 7fec5159a640 1 -- 172.21.15.135:0/2819522990 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec480017f0 con 0x7fec5406ade0 2024-03-29T11:25:38.536 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.549+0000 7fec5159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:38.536 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.549+0000 7fec5159a640 1 -- 172.21.15.135:0/2819522990 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fec48001c00 con 0x7fec5406ade0 2024-03-29T11:25:38.536 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.549+0000 7fec5159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:38.536 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.549+0000 7fec5159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:38.536 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.549+0000 7fec5159a640 10 monclient: dump: 2024-03-29T11:25:38.536 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:38.536 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:38.536 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:38.536 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:38.536 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:38.536 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:38.536 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:38.536 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:38.537 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:38.537 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:38.537 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.549+0000 7fec58e7b640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:38.537 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.549+0000 7fec58e7b640 4 set_mon_vals no callback set 2024-03-29T11:25:38.537 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.550+0000 7fec58e7b640 10 monclient: shutdown 2024-03-29T11:25:38.537 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.550+0000 7fec58e7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:38.537 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.550+0000 7fec58e7b640 1 -- 172.21.15.135:0/2819522990 >> v1:172.21.15.135:6790/0 conn(0x7fec5406ade0 legacy=0x7fec541499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:38.537 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.550+0000 7fec58e7b640 1 -- 172.21.15.135:0/2819522990 shutdown_connections 2024-03-29T11:25:38.537 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.550+0000 7fec58e7b640 1 -- 172.21.15.135:0/2819522990 >> 172.21.15.135:0/2819522990 conn(0x7fec54067470 msgr2=0x7fec540b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:38.537 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.550+0000 7fec58e7b640 1 -- 172.21.15.135:0/2819522990 shutdown_connections 2024-03-29T11:25:38.537 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.550+0000 7fec58e7b640 1 -- 172.21.15.135:0/2819522990 wait complete. 2024-03-29T11:25:38.537 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.552+0000 7fec58e7b640 10 monclient: build_initial_monmap 2024-03-29T11:25:38.537 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.552+0000 7fec58e7b640 10 monclient: monmap: 2024-03-29T11:25:38.537 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:25:38.537 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:38.538 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:25:38.538 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:25:38.538 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:38.538 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:38.538 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:25:38.538 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:25:38.538 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:25:38.538 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:38.543 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.552+0000 7fec58e7b640 1 librados: starting msgr at 2024-03-29T11:25:38.543 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.552+0000 7fec58e7b640 1 librados: starting objecter 2024-03-29T11:25:38.544 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.553+0000 7fec58e7b640 1 Processor -- start 2024-03-29T11:25:38.544 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.553+0000 7fec58e7b640 1 -- start start 2024-03-29T11:25:38.544 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.553+0000 7fec58e7b640 1 librados: setting wanted keys 2024-03-29T11:25:38.544 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.553+0000 7fec58e7b640 1 librados: calling monclient init 2024-03-29T11:25:38.544 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.553+0000 7fec58e7b640 10 monclient: init 2024-03-29T11:25:38.544 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.554+0000 7fec58e7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:38.544 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.554+0000 7fec58e7b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:38.544 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.554+0000 7fec58e7b640 10 monclient(hunting): picked mon.noname-b con 0x7fec5406ade0 addr v1:172.21.15.194:6789/0 2024-03-29T11:25:38.544 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.554+0000 7fec58e7b640 10 monclient(hunting): picked mon.noname-c con 0x7fec540721c0 addr v1:172.21.15.135:6790/0 2024-03-29T11:25:38.544 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.554+0000 7fec51d9b640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7fec540721c0 0x7fec541de860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:49386/0 (socket says 172.21.15.135:49386) 2024-03-29T11:25:38.545 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.554+0000 7fec51d9b640 1 -- 172.21.15.135:0/61069031 learned_addr learned my addr 172.21.15.135:0/61069031 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:25:38.545 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.554+0000 7fec58e7b640 10 monclient(hunting): picked mon.noname-a con 0x7fec540b06f0 addr v1:172.21.15.135:6789/0 2024-03-29T11:25:38.545 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.554+0000 7fec58e7b640 1 -- 172.21.15.135:0/61069031 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec540ab910 con 0x7fec540b06f0 2024-03-29T11:25:38.545 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.554+0000 7fec42ffd640 10 client.?.objecter ms_handle_connect 0x7fec540721c0 2024-03-29T11:25:38.545 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.554+0000 7fec42ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:38.545 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.554+0000 7fec42ffd640 10 client.?.objecter ms_handle_connect 0x7fec540b06f0 2024-03-29T11:25:38.545 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.554+0000 7fec42ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:38.545 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.554+0000 7fec42ffd640 10 client.?.objecter ms_handle_connect 0x7fec5406ade0 2024-03-29T11:25:38.545 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.554+0000 7fec42ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:38.545 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.554+0000 7fec58e7b640 1 -- 172.21.15.135:0/61069031 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec540ac080 con 0x7fec5406ade0 2024-03-29T11:25:38.545 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.554+0000 7fec58e7b640 1 -- 172.21.15.135:0/61069031 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec540aaf50 con 0x7fec540721c0 2024-03-29T11:25:38.545 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.554+0000 7fec58e7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.554+0000 7fec58e7b640 10 monclient(hunting): authenticate will time out at 845.922915s 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.555+0000 7fec42ffd640 1 -- 172.21.15.135:0/61069031 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359664120 0 0) 0x7fec540ac080 con 0x7fec5406ade0 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.555+0000 7fec42ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.555+0000 7fec42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.555+0000 7fec42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.555+0000 7fec42ffd640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.555+0000 7fec42ffd640 1 -- 172.21.15.135:0/61069031 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec34004eb0 con 0x7fec5406ade0 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.555+0000 7fec42ffd640 1 -- 172.21.15.135:0/61069031 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3178264649 0 0) 0x7fec540ab910 con 0x7fec540b06f0 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.555+0000 7fec42ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.555+0000 7fec42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.555+0000 7fec42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.555+0000 7fec42ffd640 10 monclient(hunting): my global_id is 4284 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.555+0000 7fec42ffd640 1 -- 172.21.15.135:0/61069031 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec540ac080 con 0x7fec540b06f0 2024-03-29T11:25:38.546 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.555+0000 7fec42ffd640 1 -- 172.21.15.135:0/61069031 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3076841916 0 0) 0x7fec540aaf50 con 0x7fec540721c0 2024-03-29T11:25:38.547 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.555+0000 7fec42ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:38.547 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.555+0000 7fec42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:38.547 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.555+0000 7fec42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:38.547 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.555+0000 7fec42ffd640 10 monclient(hunting): my global_id is 4265 2024-03-29T11:25:38.547 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.555+0000 7fec42ffd640 1 -- 172.21.15.135:0/61069031 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec540ab910 con 0x7fec540721c0 2024-03-29T11:25:38.547 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.555+0000 7fec42ffd640 1 -- 172.21.15.135:0/61069031 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 83886077 0 0) 0x7fec34004eb0 con 0x7fec5406ade0 2024-03-29T11:25:38.547 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.555+0000 7fec42ffd640 1 -- 172.21.15.135:0/61069031 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec540aaf50 con 0x7fec5406ade0 2024-03-29T11:25:38.547 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.555+0000 7fec42ffd640 1 -- 172.21.15.135:0/61069031 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fec48001b40 con 0x7fec5406ade0 2024-03-29T11:25:38.547 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.555+0000 7fec42ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:38.547 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.555+0000 7fec42ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:38.547 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.555+0000 7fec42ffd640 10 monclient(hunting): dump: 2024-03-29T11:25:38.547 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:38.547 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:38.547 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.555+0000 7fec42ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.555+0000 7fec42ffd640 1 -- 172.21.15.135:0/61069031 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1766992558 0 0) 0x7fec540ac080 con 0x7fec540b06f0 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.556+0000 7fec42ffd640 1 -- 172.21.15.135:0/61069031 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec34004eb0 con 0x7fec540b06f0 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.556+0000 7fec42ffd640 1 -- 172.21.15.135:0/61069031 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fec4c001b60 con 0x7fec540b06f0 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.556+0000 7fec42ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.556+0000 7fec42ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.556+0000 7fec42ffd640 10 monclient(hunting): dump: 2024-03-29T11:25:38.548 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:38.549 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:38.549 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:38.549 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:38.549 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:38.549 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:38.549 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:38.549 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:38.549 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:38.549 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:38.549 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.556+0000 7fec42ffd640 1 -- 172.21.15.135:0/61069031 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3591967112 0 0) 0x7fec540aaf50 con 0x7fec5406ade0 2024-03-29T11:25:38.549 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.556+0000 7fec42ffd640 1 -- 172.21.15.135:0/61069031 >> v1:172.21.15.135:6790/0 conn(0x7fec540721c0 legacy=0x7fec541de860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:38.549 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.556+0000 7fec42ffd640 1 -- 172.21.15.135:0/61069031 >> v1:172.21.15.135:6789/0 conn(0x7fec540b06f0 legacy=0x7fec541e5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:38.549 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.556+0000 7fec42ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:38.549 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.556+0000 7fec42ffd640 1 monclient: found mon.b 2024-03-29T11:25:38.550 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.556+0000 7fec42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:38.550 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.556+0000 7fec42ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:25:38.550 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.556+0000 7fec42ffd640 1 -- 172.21.15.135:0/61069031 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec541461f0 con 0x7fec5406ade0 2024-03-29T11:25:38.550 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.556+0000 7fec42ffd640 10 monclient: _finish_auth 0 2024-03-29T11:25:38.550 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.556+0000 7fec42ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:38.550 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.556+0000 7fec42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:38.550 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.556+0000 7fec58e7b640 5 monclient: authenticate success, global_id 4276 2024-03-29T11:25:38.550 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.556+0000 7fec58e7b640 10 monclient: _renew_subs 2024-03-29T11:25:38.550 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.556+0000 7fec58e7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:25:38.550 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.556+0000 7fec58e7b640 1 -- 172.21.15.135:0/61069031 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec54146440 con 0x7fec5406ade0 2024-03-29T11:25:38.550 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.556+0000 7fec58e7b640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:38.550 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.556+0000 7fec58e7b640 10 monclient: _renew_subs 2024-03-29T11:25:38.550 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.556+0000 7fec58e7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:25:38.550 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.556+0000 7fec58e7b640 1 -- 172.21.15.135:0/61069031 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec541df150 con 0x7fec5406ade0 2024-03-29T11:25:38.551 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.557+0000 7fec58e7b640 1 librados: init done 2024-03-29T11:25:38.551 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.557+0000 7fec42ffd640 1 -- 172.21.15.135:0/61069031 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec480038a0 con 0x7fec5406ade0 2024-03-29T11:25:38.551 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.557+0000 7fec42ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:38.551 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.557+0000 7fec42ffd640 1 -- 172.21.15.135:0/61069031 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fec48003bb0 con 0x7fec5406ade0 2024-03-29T11:25:38.551 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.557+0000 7fec42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:38.551 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.557+0000 7fec42ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:38.551 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.557+0000 7fec42ffd640 10 monclient: dump: 2024-03-29T11:25:38.551 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:38.551 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:38.551 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:38.551 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:38.551 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:38.551 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:38.551 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:38.551 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:38.552 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:38.552 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:38.552 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.557+0000 7fec50d99640 4 set_mon_vals no callback set 2024-03-29T11:25:38.552 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.559+0000 7fec42ffd640 1 -- 172.21.15.135:0/61069031 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fec48019b80 con 0x7fec5406ade0 2024-03-29T11:25:38.552 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.559+0000 7fec58e7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:38.552 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.559+0000 7fec58e7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:38.552 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.559+0000 7fec58e7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:25:38.552 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.559+0000 7fec58e7b640 1 -- 172.21.15.135:0/61069031 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec38005080 con 0x7fec5406ade0 2024-03-29T11:25:38.553 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.560+0000 7fec51d9b640 10 monclient: get_auth_request con 0x7fec3406e050 auth_method 0 2024-03-29T11:25:38.553 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.560+0000 7fec42ffd640 1 -- 172.21.15.135:0/61069031 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7fec48083950 con 0x7fec5406ade0 2024-03-29T11:25:38.553 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.560+0000 7fec42ffd640 10 client.4276.objecter ms_dispatch 0x7fec54000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:38.553 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.560+0000 7fec42ffd640 3 client.4276.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:38.553 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.560+0000 7fec42ffd640 3 client.4276.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:38.553 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.560+0000 7fec42ffd640 20 client.4276.objecter dump_active .. 0 homeless 2024-03-29T11:25:38.553 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.560+0000 7fec42ffd640 10 client.4276.objecter ms_handle_connect 0x7fec3406e050 2024-03-29T11:25:38.554 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.567+0000 7fec42ffd640 1 -- 172.21.15.135:0/61069031 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fec4804f230 con 0x7fec5406ade0 2024-03-29T11:25:38.554 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.567+0000 7fec42ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:38.554 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.567+0000 7fec42ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:38.926 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.942+0000 7fec58e7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:38.926 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.942+0000 7fec58e7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:38.927 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.942+0000 7fec58e7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:25:38.927 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.942+0000 7fec58e7b640 1 -- 172.21.15.135:0/61069031 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec38005330 con 0x7fec5406ade0 2024-03-29T11:25:38.927 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:25:38.928 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:38.928 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.943+0000 7fec42ffd640 1 -- 172.21.15.135:0/61069031 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7fec48051160 con 0x7fec5406ade0 2024-03-29T11:25:38.928 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.943+0000 7fec42ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:38.928 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.943+0000 7fec42ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:38.932 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.947+0000 7fec58e7b640 10 librados: watch_flush enter 2024-03-29T11:25:38.932 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.947+0000 7fec58e7b640 10 librados: watch_flush exit 2024-03-29T11:25:38.932 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.947+0000 7fec58e7b640 20 client.4276.objecter shutdown clearing up homeless session... 2024-03-29T11:25:38.932 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.947+0000 7fec58e7b640 10 client.4276.objecter successfully canceled tick 2024-03-29T11:25:38.932 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.948+0000 7fec58e7b640 1 -- 172.21.15.135:0/61069031 >> v1:172.21.15.135:6816/3469875285 conn(0x7fec3406e050 legacy=0x7fec34070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:38.932 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.948+0000 7fec58e7b640 10 monclient: shutdown 2024-03-29T11:25:38.932 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.948+0000 7fec58e7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:38.932 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.948+0000 7fec58e7b640 1 -- 172.21.15.135:0/61069031 >> v1:172.21.15.194:6789/0 conn(0x7fec5406ade0 legacy=0x7fec540ab200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:38.932 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.948+0000 7fec58e7b640 1 -- 172.21.15.135:0/61069031 shutdown_connections 2024-03-29T11:25:38.932 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.948+0000 7fec58e7b640 1 -- 172.21.15.135:0/61069031 >> 172.21.15.135:0/61069031 conn(0x7fec54067470 msgr2=0x7fec540b3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:38.932 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.948+0000 7fec58e7b640 1 -- 172.21.15.135:0/61069031 shutdown_connections 2024-03-29T11:25:38.932 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.948+0000 7fec58e7b640 1 -- 172.21.15.135:0/61069031 wait complete. 2024-03-29T11:25:38.933 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:38.948+0000 7fec58e7b640 1 librados: shutdown 2024-03-29T11:25:38.947 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:45.754 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:45.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.922+0000 7f5687539640 10 monclient: get_monmap_and_config 2024-03-29T11:25:45.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.922+0000 7f5687539640 10 monclient: build_initial_monmap 2024-03-29T11:25:45.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.922+0000 7f5687539640 10 monclient: monmap: 2024-03-29T11:25:45.908 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:25:45.908 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:45.908 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:25:45.923809+0000 2024-03-29T11:25:45.908 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:25:45.923809+0000 2024-03-29T11:25:45.908 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:45.908 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:45.908 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:25:45.908 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:25:45.909 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:25:45.909 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:45.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.923+0000 7f5687539640 1 Processor -- start 2024-03-29T11:25:45.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.923+0000 7f5687539640 1 -- start start 2024-03-29T11:25:45.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.923+0000 7f5687539640 10 monclient: init 2024-03-29T11:25:45.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.924+0000 7f5687539640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:45.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.924+0000 7f5687539640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:45.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.924+0000 7f5687539640 10 monclient(hunting): picked mon.noname-c con 0x7f56800b0770 addr v1:172.21.15.194:6789/0 2024-03-29T11:25:45.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.924+0000 7f5687539640 10 monclient(hunting): picked mon.noname-b con 0x7f568014a700 addr v1:172.21.15.135:6790/0 2024-03-29T11:25:45.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.924+0000 7f5687539640 10 monclient(hunting): picked mon.noname-a con 0x7f56800ac5a0 addr v1:172.21.15.135:6789/0 2024-03-29T11:25:45.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.924+0000 7f5687539640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56800aba70 con 0x7f56800ac5a0 2024-03-29T11:25:45.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.924+0000 7f5687539640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56800aecb0 con 0x7f56800b0770 2024-03-29T11:25:45.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.924+0000 7f5687539640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56800af150 con 0x7f568014a700 2024-03-29T11:25:45.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.924+0000 7f5687539640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:45.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.924+0000 7f5687539640 10 monclient(hunting): authenticate will time out at 853.293704s 2024-03-29T11:25:45.912 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.927+0000 7f5685ad6640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f56800ac5a0 0x7f5680151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:58609/0 (socket says 172.21.15.135:58609) 2024-03-29T11:25:45.912 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.927+0000 7f5685ad6640 1 -- 172.21.15.135:0/3870629403 learned_addr learned my addr 172.21.15.135:0/3870629403 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:25:45.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.927+0000 7f5677fff640 1 -- 172.21.15.135:0/3870629403 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3718464624 0 0) 0x7f56800af150 con 0x7f568014a700 2024-03-29T11:25:45.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.927+0000 7f5677fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:45.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.927+0000 7f5677fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:45.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.927+0000 7f5677fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:45.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.927+0000 7f5677fff640 10 monclient(hunting): my global_id is 4268 2024-03-29T11:25:45.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.927+0000 7f5677fff640 1 -- 172.21.15.135:0/3870629403 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f566c003520 con 0x7f568014a700 2024-03-29T11:25:45.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.928+0000 7f5677fff640 1 -- 172.21.15.135:0/3870629403 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2940907584 0 0) 0x7f566c003520 con 0x7f568014a700 2024-03-29T11:25:45.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.928+0000 7f5677fff640 1 -- 172.21.15.135:0/3870629403 >> v1:172.21.15.194:6789/0 conn(0x7f56800b0770 legacy=0x7f5680149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:45.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.928+0000 7f5677fff640 1 -- 172.21.15.135:0/3870629403 >> v1:172.21.15.135:6789/0 conn(0x7f56800ac5a0 legacy=0x7f5680151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:45.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.928+0000 7f5677fff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:45.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.928+0000 7f5677fff640 1 monclient: found mon.noname-b 2024-03-29T11:25:45.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.928+0000 7f5677fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:45.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.928+0000 7f5677fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:25:45.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.928+0000 7f5677fff640 1 -- 172.21.15.135:0/3870629403 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56800ac9d0 con 0x7f568014a700 2024-03-29T11:25:45.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.928+0000 7f5677fff640 10 monclient: _finish_auth 0 2024-03-29T11:25:45.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.928+0000 7f5677fff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:45.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.928+0000 7f5677fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:45.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.928+0000 7f5687539640 5 monclient: authenticate success, global_id 4268 2024-03-29T11:25:45.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.928+0000 7f5687539640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:45.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.928+0000 7f5677fff640 1 -- 172.21.15.135:0/3870629403 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f567c001690 con 0x7f568014a700 2024-03-29T11:25:45.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.928+0000 7f5677fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:45.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.928+0000 7f5677fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:45.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.928+0000 7f5677fff640 10 monclient: dump: 2024-03-29T11:25:45.915 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:45.915 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:45.916 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:45.916 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:45.916 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:45.916 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:45.916 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:45.916 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:45.916 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:45.916 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:45.916 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.928+0000 7f5687539640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:45.916 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.928+0000 7f5677fff640 1 -- 172.21.15.135:0/3870629403 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f567c0018b0 con 0x7f568014a700 2024-03-29T11:25:45.916 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.928+0000 7f5677fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:45.918 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.928+0000 7f5687539640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:45.918 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.928+0000 7f5687539640 4 set_mon_vals no callback set 2024-03-29T11:25:45.918 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.929+0000 7f5677fff640 1 -- 172.21.15.135:0/3870629403 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f567c001c40 con 0x7f568014a700 2024-03-29T11:25:45.918 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.929+0000 7f5677fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:45.918 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.929+0000 7f5677fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:45.918 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.929+0000 7f5677fff640 10 monclient: dump: 2024-03-29T11:25:45.918 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:45.918 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:45.918 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:45.918 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:45.919 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:45.919 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:45.919 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:45.919 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:45.919 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:45.919 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:45.919 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.929+0000 7f5687539640 10 monclient: shutdown 2024-03-29T11:25:45.919 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.929+0000 7f5687539640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:45.920 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.929+0000 7f5687539640 1 -- 172.21.15.135:0/3870629403 >> v1:172.21.15.135:6790/0 conn(0x7f568014a700 legacy=0x7f56800abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:45.920 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.929+0000 7f5687539640 1 -- 172.21.15.135:0/3870629403 shutdown_connections 2024-03-29T11:25:45.920 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.929+0000 7f5687539640 1 -- 172.21.15.135:0/3870629403 >> 172.21.15.135:0/3870629403 conn(0x7f568006b660 msgr2=0x7f568006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:45.920 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.929+0000 7f5687539640 1 -- 172.21.15.135:0/3870629403 shutdown_connections 2024-03-29T11:25:45.920 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.929+0000 7f5687539640 1 -- 172.21.15.135:0/3870629403 wait complete. 2024-03-29T11:25:45.920 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.930+0000 7f5687539640 10 monclient: build_initial_monmap 2024-03-29T11:25:45.920 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.930+0000 7f5687539640 10 monclient: monmap: 2024-03-29T11:25:45.920 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:25:45.920 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:45.920 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:25:45.920 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:25:45.920 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:45.920 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:45.920 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:25:45.920 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:25:45.921 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:25:45.921 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:45.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.931+0000 7f5687539640 1 librados: starting msgr at 2024-03-29T11:25:45.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.931+0000 7f5687539640 1 librados: starting objecter 2024-03-29T11:25:45.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.931+0000 7f5687539640 1 Processor -- start 2024-03-29T11:25:45.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.931+0000 7f5687539640 1 -- start start 2024-03-29T11:25:45.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.931+0000 7f5687539640 1 librados: setting wanted keys 2024-03-29T11:25:45.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.931+0000 7f5687539640 1 librados: calling monclient init 2024-03-29T11:25:45.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.931+0000 7f5687539640 10 monclient: init 2024-03-29T11:25:45.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.932+0000 7f5687539640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:45.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.932+0000 7f5687539640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:45.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.932+0000 7f5687539640 10 monclient(hunting): picked mon.noname-b con 0x7f56800ac5a0 addr v1:172.21.15.194:6789/0 2024-03-29T11:25:45.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.932+0000 7f5687539640 10 monclient(hunting): picked mon.noname-c con 0x7f56800b0770 addr v1:172.21.15.135:6790/0 2024-03-29T11:25:45.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.932+0000 7f5687539640 10 monclient(hunting): picked mon.noname-a con 0x7f568014a700 addr v1:172.21.15.135:6789/0 2024-03-29T11:25:45.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.932+0000 7f5687539640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5680128540 con 0x7f568014a700 2024-03-29T11:25:45.922 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.932+0000 7f5687539640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56801287b0 con 0x7f56800ac5a0 2024-03-29T11:25:45.922 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.932+0000 7f5687539640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5680126510 con 0x7f56800b0770 2024-03-29T11:25:45.922 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.932+0000 7f5687539640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:45.922 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.932+0000 7f5687539640 10 monclient(hunting): authenticate will time out at 853.301255s 2024-03-29T11:25:45.932 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.935+0000 7f5685ad6640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f568014a700 0x7f5680127140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:58707/0 (socket says 172.21.15.135:58707) 2024-03-29T11:25:45.933 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.935+0000 7f5685ad6640 1 -- 172.21.15.135:0/1199043246 learned_addr learned my addr 172.21.15.135:0/1199043246 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:25:45.933 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.935+0000 7f5675ffb640 10 client.?.objecter ms_handle_connect 0x7f568014a700 2024-03-29T11:25:45.933 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.935+0000 7f5675ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:45.933 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.935+0000 7f5675ffb640 1 -- 172.21.15.135:0/1199043246 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024707108 0 0) 0x7f5680128540 con 0x7f568014a700 2024-03-29T11:25:45.933 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.935+0000 7f5675ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:45.933 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.936+0000 7f5675ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:45.934 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.936+0000 7f5675ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:45.934 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.936+0000 7f5675ffb640 10 monclient(hunting): my global_id is 4287 2024-03-29T11:25:45.934 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.936+0000 7f5675ffb640 1 -- 172.21.15.135:0/1199043246 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f566c003160 con 0x7f568014a700 2024-03-29T11:25:45.934 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.936+0000 7f5675ffb640 10 client.?.objecter ms_handle_connect 0x7f56800ac5a0 2024-03-29T11:25:45.934 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.936+0000 7f5675ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:45.934 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.936+0000 7f5675ffb640 1 -- 172.21.15.135:0/1199043246 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058249216 0 0) 0x7f56801287b0 con 0x7f56800ac5a0 2024-03-29T11:25:45.935 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.936+0000 7f5675ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:45.935 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.936+0000 7f5675ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:45.935 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.936+0000 7f5675ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:45.935 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.936+0000 7f5675ffb640 10 monclient(hunting): my global_id is 4282 2024-03-29T11:25:45.935 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.936+0000 7f5675ffb640 1 -- 172.21.15.135:0/1199043246 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5680128540 con 0x7f56800ac5a0 2024-03-29T11:25:45.935 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.936+0000 7f5675ffb640 1 -- 172.21.15.135:0/1199043246 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2476839720 0 0) 0x7f566c003160 con 0x7f568014a700 2024-03-29T11:25:45.936 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.936+0000 7f5675ffb640 1 -- 172.21.15.135:0/1199043246 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56801287b0 con 0x7f568014a700 2024-03-29T11:25:45.936 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.936+0000 7f5675ffb640 1 -- 172.21.15.135:0/1199043246 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f5678001bb0 con 0x7f568014a700 2024-03-29T11:25:45.936 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.936+0000 7f5675ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:45.936 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.936+0000 7f5675ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:45.936 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.936+0000 7f5675ffb640 10 monclient(hunting): dump: 2024-03-29T11:25:45.936 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:45.937 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:45.937 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:45.937 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:45.937 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:45.937 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:45.937 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:45.938 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:45.938 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:45.938 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:45.938 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.936+0000 7f5675ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:45.938 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.936+0000 7f5675ffb640 1 -- 172.21.15.135:0/1199043246 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2709031399 0 0) 0x7f56801287b0 con 0x7f568014a700 2024-03-29T11:25:45.939 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.936+0000 7f5675ffb640 1 -- 172.21.15.135:0/1199043246 >> v1:172.21.15.135:6790/0 conn(0x7f56800b0770 legacy=0x7f5680126a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:45.939 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.937+0000 7f5675ffb640 1 -- 172.21.15.135:0/1199043246 >> v1:172.21.15.194:6789/0 conn(0x7f56800ac5a0 legacy=0x7f56801450c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:45.939 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.938+0000 7f5675ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:45.939 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.938+0000 7f5675ffb640 1 monclient: found mon.a 2024-03-29T11:25:45.939 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.938+0000 7f5675ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:45.939 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.938+0000 7f5675ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:25:45.939 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.938+0000 7f5675ffb640 1 -- 172.21.15.135:0/1199043246 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5680126150 con 0x7f568014a700 2024-03-29T11:25:45.939 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.938+0000 7f5675ffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:45.939 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.938+0000 7f5675ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:45.939 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.938+0000 7f5675ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:45.940 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.938+0000 7f5687539640 5 monclient: authenticate success, global_id 4287 2024-03-29T11:25:45.940 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.938+0000 7f5687539640 10 monclient: _renew_subs 2024-03-29T11:25:45.940 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.938+0000 7f5687539640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:25:45.940 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.938+0000 7f5687539640 1 -- 172.21.15.135:0/1199043246 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56801dfa80 con 0x7f568014a700 2024-03-29T11:25:45.940 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.938+0000 7f5687539640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:45.940 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.938+0000 7f5687539640 10 monclient: _renew_subs 2024-03-29T11:25:45.940 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.938+0000 7f5687539640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:25:45.940 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.938+0000 7f5687539640 1 -- 172.21.15.135:0/1199043246 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f56801df240 con 0x7f568014a700 2024-03-29T11:25:45.940 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.938+0000 7f5687539640 1 librados: init done 2024-03-29T11:25:45.940 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.939+0000 7f5675ffb640 1 -- 172.21.15.135:0/1199043246 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5678001d10 con 0x7f568014a700 2024-03-29T11:25:45.940 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.939+0000 7f5675ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:45.940 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.939+0000 7f5675ffb640 1 -- 172.21.15.135:0/1199043246 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f5678004d00 con 0x7f568014a700 2024-03-29T11:25:45.940 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.939+0000 7f5675ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:45.940 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.939+0000 7f5675ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:45.940 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.939+0000 7f5675ffb640 10 monclient: dump: 2024-03-29T11:25:45.941 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:45.941 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:45.941 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:45.941 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:45.941 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:45.941 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:45.941 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:45.941 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:45.941 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:45.941 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:45.941 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.939+0000 7f5676ffd640 4 set_mon_vals no callback set 2024-03-29T11:25:45.941 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.940+0000 7f5687539640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:45.941 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.940+0000 7f5675ffb640 1 -- 172.21.15.135:0/1199043246 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f5678004f40 con 0x7f568014a700 2024-03-29T11:25:45.941 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.940+0000 7f5687539640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:45.942 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.940+0000 7f5687539640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:25:45.942 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.940+0000 7f5687539640 1 -- 172.21.15.135:0/1199043246 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5660005080 con 0x7f568014a700 2024-03-29T11:25:45.942 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.941+0000 7f56852d5640 10 monclient: get_auth_request con 0x7f566c06dc50 auth_method 0 2024-03-29T11:25:45.942 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.941+0000 7f5675ffb640 1 -- 172.21.15.135:0/1199043246 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f5678084e00 con 0x7f568014a700 2024-03-29T11:25:45.942 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.941+0000 7f5675ffb640 10 client.4287.objecter ms_dispatch 0x7f5680000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:45.942 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.941+0000 7f5675ffb640 3 client.4287.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:45.942 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.941+0000 7f5675ffb640 3 client.4287.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:45.942 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.941+0000 7f5675ffb640 20 client.4287.objecter dump_active .. 0 homeless 2024-03-29T11:25:45.942 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.941+0000 7f5675ffb640 10 client.4287.objecter ms_handle_connect 0x7f566c06dc50 2024-03-29T11:25:45.943 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.950+0000 7f5675ffb640 1 -- 172.21.15.135:0/1199043246 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f56780506e0 con 0x7f568014a700 2024-03-29T11:25:45.943 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.950+0000 7f5675ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:45.943 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:45.950+0000 7f5675ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:46.267 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:46.282+0000 7f5687539640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:46.268 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:46.282+0000 7f5687539640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:46.268 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:46.282+0000 7f5687539640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:25:46.268 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:46.282+0000 7f5687539640 1 -- 172.21.15.135:0/1199043246 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5660005330 con 0x7f568014a700 2024-03-29T11:25:46.268 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:46.283+0000 7f5675ffb640 1 -- 172.21.15.135:0/1199043246 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f5678052610 con 0x7f568014a700 2024-03-29T11:25:46.269 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:46.283+0000 7f5675ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:46.269 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:46.283+0000 7f5675ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:46.270 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:25:46.270 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:46.274 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:46.288+0000 7f5687539640 10 librados: watch_flush enter 2024-03-29T11:25:46.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:46.289+0000 7f5687539640 10 librados: watch_flush exit 2024-03-29T11:25:46.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:46.289+0000 7f5687539640 20 client.4287.objecter shutdown clearing up homeless session... 2024-03-29T11:25:46.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:46.289+0000 7f5687539640 10 client.4287.objecter successfully canceled tick 2024-03-29T11:25:46.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:46.290+0000 7f5687539640 1 -- 172.21.15.135:0/1199043246 >> v1:172.21.15.135:6816/3469875285 conn(0x7f566c06dc50 legacy=0x7f566c070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:46.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:46.290+0000 7f5687539640 10 monclient: shutdown 2024-03-29T11:25:46.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:46.290+0000 7f5687539640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:46.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:46.290+0000 7f5687539640 1 -- 172.21.15.135:0/1199043246 >> v1:172.21.15.135:6789/0 conn(0x7f568014a700 legacy=0x7f5680127140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:46.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:46.290+0000 7f5687539640 1 -- 172.21.15.135:0/1199043246 shutdown_connections 2024-03-29T11:25:46.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:46.290+0000 7f5687539640 1 -- 172.21.15.135:0/1199043246 >> 172.21.15.135:0/1199043246 conn(0x7f568006b660 msgr2=0x7f568006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:46.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:46.290+0000 7f5687539640 1 -- 172.21.15.135:0/1199043246 shutdown_connections 2024-03-29T11:25:46.276 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:46.290+0000 7f5687539640 1 -- 172.21.15.135:0/1199043246 wait complete. 2024-03-29T11:25:46.276 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:46.290+0000 7f5687539640 1 librados: shutdown 2024-03-29T11:25:46.293 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:53.100 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:53.246 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.260+0000 7fc393001640 10 monclient: get_monmap_and_config 2024-03-29T11:25:53.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.260+0000 7fc393001640 10 monclient: build_initial_monmap 2024-03-29T11:25:53.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.260+0000 7fc393001640 10 monclient: monmap: 2024-03-29T11:25:53.247 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:25:53.247 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:53.247 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:25:53.261699+0000 2024-03-29T11:25:53.247 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:25:53.261699+0000 2024-03-29T11:25:53.247 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:53.247 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:53.247 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:25:53.247 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:25:53.247 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:25:53.248 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:53.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.261+0000 7fc393001640 1 Processor -- start 2024-03-29T11:25:53.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.262+0000 7fc393001640 1 -- start start 2024-03-29T11:25:53.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.262+0000 7fc393001640 10 monclient: init 2024-03-29T11:25:53.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.262+0000 7fc393001640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:53.249 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.262+0000 7fc393001640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:53.249 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.262+0000 7fc393001640 10 monclient(hunting): picked mon.noname-a con 0x7fc38c0b3f30 addr v1:172.21.15.135:6789/0 2024-03-29T11:25:53.249 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.262+0000 7fc393001640 10 monclient(hunting): picked mon.noname-c con 0x7fc38c0a9750 addr v1:172.21.15.194:6789/0 2024-03-29T11:25:53.249 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.263+0000 7fc393001640 10 monclient(hunting): picked mon.noname-b con 0x7fc38c13b9d0 addr v1:172.21.15.135:6790/0 2024-03-29T11:25:53.249 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.263+0000 7fc393001640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc38c142490 con 0x7fc38c0b3f30 2024-03-29T11:25:53.249 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.263+0000 7fc393001640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc38c0ac130 con 0x7fc38c0a9750 2024-03-29T11:25:53.249 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.263+0000 7fc393001640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc38c0ac410 con 0x7fc38c13b9d0 2024-03-29T11:25:53.249 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.263+0000 7fc393001640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:53.249 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.263+0000 7fc393001640 10 monclient(hunting): authenticate will time out at 860.632115s 2024-03-29T11:25:53.249 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.263+0000 7fc3917fe640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fc38c0a9750 0x7fc38c13b2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:42312/0 (socket says 172.21.15.135:42312) 2024-03-29T11:25:53.249 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.263+0000 7fc3917fe640 1 -- 172.21.15.135:0/3024599802 learned_addr learned my addr 172.21.15.135:0/3024599802 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:25:53.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.265+0000 7fc390ffd640 1 -- 172.21.15.135:0/3024599802 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269423560 0 0) 0x7fc38c0ac130 con 0x7fc38c0a9750 2024-03-29T11:25:53.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.265+0000 7fc390ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:53.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.265+0000 7fc390ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:53.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.265+0000 7fc390ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:53.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.265+0000 7fc390ffd640 10 monclient(hunting): my global_id is 4285 2024-03-29T11:25:53.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.265+0000 7fc390ffd640 1 -- 172.21.15.135:0/3024599802 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc378003520 con 0x7fc38c0a9750 2024-03-29T11:25:53.252 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.266+0000 7fc390ffd640 1 -- 172.21.15.135:0/3024599802 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 560421533 0 0) 0x7fc378003520 con 0x7fc38c0a9750 2024-03-29T11:25:53.252 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.266+0000 7fc390ffd640 1 -- 172.21.15.135:0/3024599802 >> v1:172.21.15.135:6790/0 conn(0x7fc38c13b9d0 legacy=0x7fc38c1428f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:53.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.267+0000 7fc390ffd640 1 -- 172.21.15.135:0/3024599802 >> v1:172.21.15.135:6789/0 conn(0x7fc38c0b3f30 legacy=0x7fc38c141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:53.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.267+0000 7fc390ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:53.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.267+0000 7fc390ffd640 1 monclient: found mon.noname-c 2024-03-29T11:25:53.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.267+0000 7fc390ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:53.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.267+0000 7fc390ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:25:53.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.267+0000 7fc390ffd640 1 -- 172.21.15.135:0/3024599802 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc38c0afa40 con 0x7fc38c0a9750 2024-03-29T11:25:53.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.267+0000 7fc390ffd640 10 monclient: _finish_auth 0 2024-03-29T11:25:53.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.267+0000 7fc390ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:53.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.267+0000 7fc390ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:53.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.267+0000 7fc390ffd640 1 -- 172.21.15.135:0/3024599802 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fc388001690 con 0x7fc38c0a9750 2024-03-29T11:25:53.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.267+0000 7fc393001640 5 monclient: authenticate success, global_id 4285 2024-03-29T11:25:53.255 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.267+0000 7fc390ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:53.255 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.267+0000 7fc390ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:53.255 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.267+0000 7fc390ffd640 10 monclient: dump: 2024-03-29T11:25:53.255 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:53.255 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:53.255 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:53.255 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:53.255 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:53.255 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:53.255 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:53.255 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:53.255 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:53.255 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:53.255 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.267+0000 7fc393001640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:53.256 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.268+0000 7fc390ffd640 1 -- 172.21.15.135:0/3024599802 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3880017f0 con 0x7fc38c0a9750 2024-03-29T11:25:53.256 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.268+0000 7fc390ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:53.256 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.268+0000 7fc390ffd640 1 -- 172.21.15.135:0/3024599802 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fc388001c40 con 0x7fc38c0a9750 2024-03-29T11:25:53.256 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.268+0000 7fc390ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:53.256 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.268+0000 7fc390ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:53.256 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.268+0000 7fc390ffd640 10 monclient: dump: 2024-03-29T11:25:53.256 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:53.256 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:53.256 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:53.256 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:53.256 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:53.256 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:53.256 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:53.256 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:53.256 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:53.257 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:53.257 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.268+0000 7fc393001640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:53.257 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.268+0000 7fc393001640 4 set_mon_vals no callback set 2024-03-29T11:25:53.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.269+0000 7fc393001640 10 monclient: shutdown 2024-03-29T11:25:53.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.269+0000 7fc393001640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:53.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.269+0000 7fc393001640 1 -- 172.21.15.135:0/3024599802 >> v1:172.21.15.194:6789/0 conn(0x7fc38c0a9750 legacy=0x7fc38c13b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:53.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.270+0000 7fc393001640 1 -- 172.21.15.135:0/3024599802 shutdown_connections 2024-03-29T11:25:53.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.270+0000 7fc393001640 1 -- 172.21.15.135:0/3024599802 >> 172.21.15.135:0/3024599802 conn(0x7fc38c0673e0 msgr2=0x7fc38c0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:53.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.270+0000 7fc393001640 1 -- 172.21.15.135:0/3024599802 shutdown_connections 2024-03-29T11:25:53.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.270+0000 7fc393001640 1 -- 172.21.15.135:0/3024599802 wait complete. 2024-03-29T11:25:53.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.271+0000 7fc393001640 10 monclient: build_initial_monmap 2024-03-29T11:25:53.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.271+0000 7fc393001640 10 monclient: monmap: 2024-03-29T11:25:53.259 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:25:53.259 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:53.259 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:25:53.259 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:25:53.259 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:53.260 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:53.260 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:25:53.260 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:25:53.260 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:25:53.260 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:53.260 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.271+0000 7fc393001640 1 librados: starting msgr at 2024-03-29T11:25:53.260 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.271+0000 7fc393001640 1 librados: starting objecter 2024-03-29T11:25:53.260 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.272+0000 7fc393001640 1 Processor -- start 2024-03-29T11:25:53.260 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.272+0000 7fc393001640 1 -- start start 2024-03-29T11:25:53.260 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.272+0000 7fc393001640 1 librados: setting wanted keys 2024-03-29T11:25:53.260 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.272+0000 7fc393001640 1 librados: calling monclient init 2024-03-29T11:25:53.260 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.272+0000 7fc393001640 10 monclient: init 2024-03-29T11:25:53.260 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.272+0000 7fc393001640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:53.260 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.273+0000 7fc393001640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:53.261 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.275+0000 7fc393001640 10 monclient(hunting): picked mon.noname-c con 0x7fc38c0a9750 addr v1:172.21.15.135:6790/0 2024-03-29T11:25:53.261 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.275+0000 7fc393001640 10 monclient(hunting): picked mon.noname-b con 0x7fc38c0b3f30 addr v1:172.21.15.194:6789/0 2024-03-29T11:25:53.261 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.275+0000 7fc393001640 10 monclient(hunting): picked mon.noname-a con 0x7fc38c13b9d0 addr v1:172.21.15.135:6789/0 2024-03-29T11:25:53.261 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.275+0000 7fc393001640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc38c13ee60 con 0x7fc38c13b9d0 2024-03-29T11:25:53.261 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.275+0000 7fc393001640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc38c132dd0 con 0x7fc38c0b3f30 2024-03-29T11:25:53.261 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.275+0000 7fc393001640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc38c1352f0 con 0x7fc38c0a9750 2024-03-29T11:25:53.261 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.275+0000 7fc393001640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:53.262 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.275+0000 7fc393001640 10 monclient(hunting): authenticate will time out at 860.644638s 2024-03-29T11:25:53.262 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.276+0000 7fc391fff640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7fc38c0a9750 0x7fc38c13bfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:46263/0 (socket says 172.21.15.135:46263) 2024-03-29T11:25:53.262 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.276+0000 7fc391fff640 1 -- 172.21.15.135:0/1217975153 learned_addr learned my addr 172.21.15.135:0/1217975153 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:25:53.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.276+0000 7fc3827fc640 10 client.?.objecter ms_handle_connect 0x7fc38c0a9750 2024-03-29T11:25:53.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.276+0000 7fc3827fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:53.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.276+0000 7fc3827fc640 1 -- 172.21.15.135:0/1217975153 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810533877 0 0) 0x7fc38c1352f0 con 0x7fc38c0a9750 2024-03-29T11:25:53.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.276+0000 7fc3827fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:53.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.276+0000 7fc3827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:53.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.276+0000 7fc3827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:53.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.276+0000 7fc3827fc640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:25:53.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.276+0000 7fc3827fc640 1 -- 172.21.15.135:0/1217975153 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc370004eb0 con 0x7fc38c0a9750 2024-03-29T11:25:53.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.277+0000 7fc3827fc640 1 -- 172.21.15.135:0/1217975153 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1212477935 0 0) 0x7fc370004eb0 con 0x7fc38c0a9750 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.277+0000 7fc3827fc640 1 -- 172.21.15.135:0/1217975153 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc38c1352f0 con 0x7fc38c0a9750 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.277+0000 7fc3827fc640 1 -- 172.21.15.135:0/1217975153 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fc384001d50 con 0x7fc38c0a9750 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.277+0000 7fc3827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.277+0000 7fc3827fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.277+0000 7fc3827fc640 10 monclient(hunting): dump: 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:53.265 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.277+0000 7fc3827fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.277+0000 7fc3827fc640 10 client.?.objecter ms_handle_connect 0x7fc38c0b3f30 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.277+0000 7fc3827fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.278+0000 7fc3827fc640 1 -- 172.21.15.135:0/1217975153 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247325120 0 0) 0x7fc38c132dd0 con 0x7fc38c0b3f30 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.278+0000 7fc3827fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.278+0000 7fc3827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.278+0000 7fc3827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.278+0000 7fc3827fc640 10 monclient(hunting): my global_id is 4288 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.278+0000 7fc3827fc640 1 -- 172.21.15.135:0/1217975153 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc370004eb0 con 0x7fc38c0b3f30 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.278+0000 7fc3827fc640 1 -- 172.21.15.135:0/1217975153 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4033646285 0 0) 0x7fc38c1352f0 con 0x7fc38c0a9750 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.278+0000 7fc3827fc640 1 -- 172.21.15.135:0/1217975153 >> v1:172.21.15.194:6789/0 conn(0x7fc38c0b3f30 legacy=0x7fc38c13c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:53.266 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.278+0000 7fc3827fc640 1 -- 172.21.15.135:0/1217975153 >> v1:172.21.15.135:6789/0 conn(0x7fc38c13b9d0 legacy=0x7fc38c1e67a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:53.268 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.280+0000 7fc3827fc640 10 monclient: _finish_hunting 0 2024-03-29T11:25:53.268 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.280+0000 7fc3827fc640 1 monclient: found mon.c 2024-03-29T11:25:53.268 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.280+0000 7fc3827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:53.268 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.280+0000 7fc3827fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:25:53.268 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.280+0000 7fc3827fc640 1 -- 172.21.15.135:0/1217975153 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc38c135480 con 0x7fc38c0a9750 2024-03-29T11:25:53.268 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.280+0000 7fc3827fc640 10 monclient: _finish_auth 0 2024-03-29T11:25:53.268 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.280+0000 7fc3827fc640 10 monclient: _check_auth_tickets 2024-03-29T11:25:53.269 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.280+0000 7fc3827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:53.269 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.280+0000 7fc393001640 5 monclient: authenticate success, global_id 4271 2024-03-29T11:25:53.269 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.280+0000 7fc393001640 10 monclient: _renew_subs 2024-03-29T11:25:53.269 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.280+0000 7fc3827fc640 1 -- 172.21.15.135:0/1217975153 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc384002050 con 0x7fc38c0a9750 2024-03-29T11:25:53.269 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.280+0000 7fc393001640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:25:53.269 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.280+0000 7fc393001640 1 -- 172.21.15.135:0/1217975153 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc38c13da30 con 0x7fc38c0a9750 2024-03-29T11:25:53.269 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.280+0000 7fc393001640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:53.269 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.280+0000 7fc3827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:53.269 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.280+0000 7fc3827fc640 1 -- 172.21.15.135:0/1217975153 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fc384003de0 con 0x7fc38c0a9750 2024-03-29T11:25:53.269 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.280+0000 7fc3827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:53.269 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.280+0000 7fc3827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:53.269 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.280+0000 7fc3827fc640 10 monclient: dump: 2024-03-29T11:25:53.269 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:25:53.269 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:25:53.270 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:53.270 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:25:53.270 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:53.270 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:25:53.270 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:25:53.270 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:25:53.270 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:25:53.270 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:25:53.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.280+0000 7fc383fff640 4 set_mon_vals no callback set 2024-03-29T11:25:53.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.280+0000 7fc393001640 10 monclient: _renew_subs 2024-03-29T11:25:53.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.280+0000 7fc393001640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:25:53.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.280+0000 7fc393001640 1 -- 172.21.15.135:0/1217975153 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc38c13e8b0 con 0x7fc38c0a9750 2024-03-29T11:25:53.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.280+0000 7fc393001640 1 librados: init done 2024-03-29T11:25:53.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.282+0000 7fc393001640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:53.271 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.282+0000 7fc393001640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:53.271 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.282+0000 7fc393001640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:25:53.271 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.282+0000 7fc393001640 1 -- 172.21.15.135:0/1217975153 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc38c04f730 con 0x7fc38c0a9750 2024-03-29T11:25:53.271 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.285+0000 7fc3827fc640 1 -- 172.21.15.135:0/1217975153 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fc384003f40 con 0x7fc38c0a9750 2024-03-29T11:25:53.272 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.286+0000 7fc3827fc640 1 -- 172.21.15.135:0/1217975153 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7fc384084ed0 con 0x7fc38c0a9750 2024-03-29T11:25:53.272 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.286+0000 7fc3827fc640 10 client.4271.objecter ms_dispatch 0x7fc38c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:53.272 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.286+0000 7fc3827fc640 3 client.4271.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:53.272 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.286+0000 7fc3827fc640 3 client.4271.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:53.272 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.286+0000 7fc3827fc640 20 client.4271.objecter dump_active .. 0 homeless 2024-03-29T11:25:53.272 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.286+0000 7fc3917fe640 10 monclient: get_auth_request con 0x7fc37006dce0 auth_method 0 2024-03-29T11:25:53.272 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.286+0000 7fc3827fc640 10 client.4271.objecter ms_handle_connect 0x7fc37006dce0 2024-03-29T11:25:53.279 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.294+0000 7fc3827fc640 1 -- 172.21.15.135:0/1217975153 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc384050810 con 0x7fc38c0a9750 2024-03-29T11:25:53.279 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.294+0000 7fc3827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:53.279 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.294+0000 7fc3827fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:53.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.613+0000 7fc393001640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:53.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.613+0000 7fc393001640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:53.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.613+0000 7fc393001640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:25:53.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.613+0000 7fc393001640 1 -- 172.21.15.135:0/1217975153 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc38c135610 con 0x7fc38c0a9750 2024-03-29T11:25:53.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.616+0000 7fc3827fc640 1 -- 172.21.15.135:0/1217975153 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7fc384052740 con 0x7fc38c0a9750 2024-03-29T11:25:53.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.616+0000 7fc3827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:53.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.616+0000 7fc3827fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:53.602 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:25:53.602 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:53.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.620+0000 7fc393001640 10 librados: watch_flush enter 2024-03-29T11:25:53.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.620+0000 7fc393001640 10 librados: watch_flush exit 2024-03-29T11:25:53.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.620+0000 7fc393001640 20 client.4271.objecter shutdown clearing up homeless session... 2024-03-29T11:25:53.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.620+0000 7fc393001640 10 client.4271.objecter successfully canceled tick 2024-03-29T11:25:53.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.620+0000 7fc393001640 1 -- 172.21.15.135:0/1217975153 >> v1:172.21.15.135:6816/3469875285 conn(0x7fc37006dce0 legacy=0x7fc3700701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:53.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.620+0000 7fc393001640 10 monclient: shutdown 2024-03-29T11:25:53.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.620+0000 7fc393001640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:53.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.620+0000 7fc393001640 1 -- 172.21.15.135:0/1217975153 >> v1:172.21.15.135:6790/0 conn(0x7fc38c0a9750 legacy=0x7fc38c13bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:53.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.621+0000 7fc393001640 1 -- 172.21.15.135:0/1217975153 shutdown_connections 2024-03-29T11:25:53.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.621+0000 7fc393001640 1 -- 172.21.15.135:0/1217975153 >> 172.21.15.135:0/1217975153 conn(0x7fc38c0673e0 msgr2=0x7fc38c0b01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:53.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.621+0000 7fc393001640 1 -- 172.21.15.135:0/1217975153 shutdown_connections 2024-03-29T11:25:53.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.621+0000 7fc393001640 1 -- 172.21.15.135:0/1217975153 wait complete. 2024-03-29T11:25:53.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:25:53.621+0000 7fc393001640 1 librados: shutdown 2024-03-29T11:25:53.624 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:00.431 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:00.590 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.603+0000 7f724e612640 10 monclient: get_monmap_and_config 2024-03-29T11:26:00.590 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.603+0000 7f724e612640 10 monclient: build_initial_monmap 2024-03-29T11:26:00.590 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.603+0000 7f724e612640 10 monclient: monmap: 2024-03-29T11:26:00.590 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:26:00.590 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:00.591 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:26:00.605515+0000 2024-03-29T11:26:00.591 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:26:00.605515+0000 2024-03-29T11:26:00.591 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:00.591 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:00.591 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:26:00.591 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:26:00.591 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:26:00.591 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:00.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.605+0000 7f724e612640 1 Processor -- start 2024-03-29T11:26:00.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.606+0000 7f724e612640 1 -- start start 2024-03-29T11:26:00.597 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.606+0000 7f724e612640 10 monclient: init 2024-03-29T11:26:00.597 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.607+0000 7f724e612640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:00.597 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.607+0000 7f724e612640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:00.597 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.607+0000 7f724e612640 10 monclient(hunting): picked mon.noname-c con 0x7f7240034850 addr v1:172.21.15.194:6789/0 2024-03-29T11:26:00.597 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.607+0000 7f724e612640 10 monclient(hunting): picked mon.noname-b con 0x7f724002a640 addr v1:172.21.15.135:6790/0 2024-03-29T11:26:00.597 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.607+0000 7f724e612640 10 monclient(hunting): picked mon.noname-a con 0x7f72400327d0 addr v1:172.21.15.135:6789/0 2024-03-29T11:26:00.597 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.607+0000 7f724e612640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f724002c820 con 0x7f72400327d0 2024-03-29T11:26:00.597 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.607+0000 7f724e612640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f724002a120 con 0x7f7240034850 2024-03-29T11:26:00.597 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.607+0000 7f724e612640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72400c75b0 con 0x7f724002a640 2024-03-29T11:26:00.597 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.607+0000 7f724e612640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:00.597 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.607+0000 7f724e612640 10 monclient(hunting): authenticate will time out at 867.976220s 2024-03-29T11:26:00.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.613+0000 7f724d610640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f7240034850 0x7f72400c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:48501/0 (socket says 172.21.15.135:48501) 2024-03-29T11:26:00.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.613+0000 7f724d610640 1 -- 172.21.15.135:0/1532562534 learned_addr learned my addr 172.21.15.135:0/1532562534 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:26:00.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.613+0000 7f723ffff640 1 -- 172.21.15.135:0/1532562534 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2821058717 0 0) 0x7f72400c75b0 con 0x7f724002a640 2024-03-29T11:26:00.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.613+0000 7f723ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:00.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.613+0000 7f723ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:00.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.613+0000 7f723ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:00.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.613+0000 7f723ffff640 10 monclient(hunting): my global_id is 4274 2024-03-29T11:26:00.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.613+0000 7f723ffff640 1 -- 172.21.15.135:0/1532562534 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7234004630 con 0x7f724002a640 2024-03-29T11:26:00.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.613+0000 7f723ffff640 1 -- 172.21.15.135:0/1532562534 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369837347 0 0) 0x7f724002c820 con 0x7f72400327d0 2024-03-29T11:26:00.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.613+0000 7f723ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:00.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.613+0000 7f723ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:00.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.613+0000 7f723ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:00.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.613+0000 7f723ffff640 10 monclient(hunting): my global_id is 4293 2024-03-29T11:26:00.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.613+0000 7f723ffff640 1 -- 172.21.15.135:0/1532562534 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72400c75b0 con 0x7f72400327d0 2024-03-29T11:26:00.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.614+0000 7f723ffff640 1 -- 172.21.15.135:0/1532562534 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1013175452 0 0) 0x7f72400c75b0 con 0x7f72400327d0 2024-03-29T11:26:00.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.614+0000 7f723ffff640 1 -- 172.21.15.135:0/1532562534 >> v1:172.21.15.135:6790/0 conn(0x7f724002a640 legacy=0x7f72400320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.614+0000 7f723ffff640 1 -- 172.21.15.135:0/1532562534 >> v1:172.21.15.194:6789/0 conn(0x7f7240034850 legacy=0x7f72400c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:00.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.614+0000 7f723ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:00.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.614+0000 7f723ffff640 1 monclient: found mon.noname-a 2024-03-29T11:26:00.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.614+0000 7f723ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:00.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.614+0000 7f723ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.135:6789/0 2024-03-29T11:26:00.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.614+0000 7f723ffff640 1 -- 172.21.15.135:0/1532562534 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7240033110 con 0x7f72400327d0 2024-03-29T11:26:00.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.614+0000 7f723ffff640 10 monclient: _finish_auth 0 2024-03-29T11:26:00.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.614+0000 7f723ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:00.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.614+0000 7f723ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:00.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.614+0000 7f723ffff640 1 -- 172.21.15.135:0/1532562534 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f7238001610 con 0x7f72400327d0 2024-03-29T11:26:00.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.614+0000 7f724e612640 5 monclient: authenticate success, global_id 4293 2024-03-29T11:26:00.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.614+0000 7f724e612640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:00.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.614+0000 7f723ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:00.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.614+0000 7f723ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:00.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.614+0000 7f723ffff640 10 monclient: dump: 2024-03-29T11:26:00.604 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:00.604 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:00.604 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:00.604 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:00.604 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:00.604 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:00.604 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:00.604 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:00.604 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:00.604 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:00.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.614+0000 7f724e612640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:00.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.614+0000 7f723ffff640 1 -- 172.21.15.135:0/1532562534 <== mon.0 v1:172.21.15.135:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72380017e0 con 0x7f72400327d0 2024-03-29T11:26:00.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.614+0000 7f723ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:00.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.614+0000 7f723ffff640 1 -- 172.21.15.135:0/1532562534 <== mon.0 v1:172.21.15.135:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f7238001c30 con 0x7f72400327d0 2024-03-29T11:26:00.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.614+0000 7f724e612640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:00.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.614+0000 7f724e612640 4 set_mon_vals no callback set 2024-03-29T11:26:00.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.614+0000 7f723ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:00.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.614+0000 7f723ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:00.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.614+0000 7f723ffff640 10 monclient: dump: 2024-03-29T11:26:00.605 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:00.605 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:00.605 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:00.605 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:00.605 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:00.605 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:00.606 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:00.606 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:00.606 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:00.606 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:00.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.615+0000 7f724e612640 10 monclient: shutdown 2024-03-29T11:26:00.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.615+0000 7f724e612640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:00.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.615+0000 7f724e612640 1 -- 172.21.15.135:0/1532562534 >> v1:172.21.15.135:6789/0 conn(0x7f72400327d0 legacy=0x7f72400cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.615+0000 7f724e612640 1 -- 172.21.15.135:0/1532562534 shutdown_connections 2024-03-29T11:26:00.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.615+0000 7f724e612640 1 -- 172.21.15.135:0/1532562534 >> 172.21.15.135:0/1532562534 conn(0x7f72400058f0 msgr2=0x7f7240005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:00.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.618+0000 7f724e612640 1 -- 172.21.15.135:0/1532562534 shutdown_connections 2024-03-29T11:26:00.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.618+0000 7f724e612640 1 -- 172.21.15.135:0/1532562534 wait complete. 2024-03-29T11:26:00.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.619+0000 7f724e612640 10 monclient: build_initial_monmap 2024-03-29T11:26:00.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.619+0000 7f724e612640 10 monclient: monmap: 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.619+0000 7f724e612640 1 librados: starting msgr at 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.619+0000 7f724e612640 1 librados: starting objecter 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.619+0000 7f724e612640 1 Processor -- start 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.619+0000 7f724e612640 1 -- start start 2024-03-29T11:26:00.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.619+0000 7f724e612640 1 librados: setting wanted keys 2024-03-29T11:26:00.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.619+0000 7f724e612640 1 librados: calling monclient init 2024-03-29T11:26:00.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.619+0000 7f724e612640 10 monclient: init 2024-03-29T11:26:00.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.620+0000 7f724e612640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:00.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.620+0000 7f724e612640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:00.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.620+0000 7f724e612640 10 monclient(hunting): picked mon.noname-b con 0x7f724002a640 addr v1:172.21.15.194:6789/0 2024-03-29T11:26:00.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.621+0000 7f724e612640 10 monclient(hunting): picked mon.noname-c con 0x7f72400327d0 addr v1:172.21.15.135:6790/0 2024-03-29T11:26:00.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.621+0000 7f724e612640 10 monclient(hunting): picked mon.noname-a con 0x7f7240034850 addr v1:172.21.15.135:6789/0 2024-03-29T11:26:00.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.621+0000 7f724e612640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f724002d730 con 0x7f7240034850 2024-03-29T11:26:00.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.621+0000 7f724e612640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f724016d660 con 0x7f724002a640 2024-03-29T11:26:00.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.621+0000 7f724e612640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7240031b10 con 0x7f72400327d0 2024-03-29T11:26:00.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.621+0000 7f724e612640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:00.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.621+0000 7f724e612640 10 monclient(hunting): authenticate will time out at 867.990381s 2024-03-29T11:26:00.614 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.628+0000 7f724d610640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f724002a640 0x7f72400bff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:48569/0 (socket says 172.21.15.135:48569) 2024-03-29T11:26:00.614 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.628+0000 7f724d610640 1 -- 172.21.15.135:0/1666223807 learned_addr learned my addr 172.21.15.135:0/1666223807 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:26:00.614 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.629+0000 7f723dffb640 10 client.?.objecter ms_handle_connect 0x7f724002a640 2024-03-29T11:26:00.614 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.629+0000 7f723dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:00.617 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.631+0000 7f723dffb640 10 client.?.objecter ms_handle_connect 0x7f72400327d0 2024-03-29T11:26:00.617 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.631+0000 7f723dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:00.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.631+0000 7f723dffb640 1 -- 172.21.15.135:0/1666223807 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1611345959 0 0) 0x7f7240031b10 con 0x7f72400327d0 2024-03-29T11:26:00.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.631+0000 7f723dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:00.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.631+0000 7f723dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:00.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.631+0000 7f723dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:00.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.631+0000 7f723dffb640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:26:00.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.632+0000 7f723dffb640 1 -- 172.21.15.135:0/1666223807 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f722c004eb0 con 0x7f72400327d0 2024-03-29T11:26:00.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.632+0000 7f723dffb640 10 client.?.objecter ms_handle_connect 0x7f7240034850 2024-03-29T11:26:00.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.632+0000 7f723dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:00.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.632+0000 7f723dffb640 1 -- 172.21.15.135:0/1666223807 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2471726906 0 0) 0x7f722c004eb0 con 0x7f72400327d0 2024-03-29T11:26:00.619 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.632+0000 7f723dffb640 1 -- 172.21.15.135:0/1666223807 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7240031b10 con 0x7f72400327d0 2024-03-29T11:26:00.619 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.632+0000 7f723dffb640 1 -- 172.21.15.135:0/1666223807 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f724804fdd0 con 0x7f72400327d0 2024-03-29T11:26:00.619 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.632+0000 7f723dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:00.619 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.632+0000 7f723dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:00.619 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.632+0000 7f723dffb640 10 monclient(hunting): dump: 2024-03-29T11:26:00.619 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:00.620 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:00.620 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:00.620 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:00.620 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:00.620 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:00.620 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:00.620 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:00.620 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:00.620 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:00.620 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.632+0000 7f723dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:00.620 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.633+0000 7f723dffb640 1 -- 172.21.15.135:0/1666223807 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560480494 0 0) 0x7f724002d730 con 0x7f7240034850 2024-03-29T11:26:00.620 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.633+0000 7f723dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:00.620 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.633+0000 7f723dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:00.620 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.633+0000 7f723dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:00.621 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.633+0000 7f723dffb640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:26:00.621 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.633+0000 7f723dffb640 1 -- 172.21.15.135:0/1666223807 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f722c004eb0 con 0x7f7240034850 2024-03-29T11:26:00.622 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.634+0000 7f723dffb640 1 -- 172.21.15.135:0/1666223807 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1617122529 0 0) 0x7f722c004eb0 con 0x7f7240034850 2024-03-29T11:26:00.622 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.634+0000 7f723dffb640 1 -- 172.21.15.135:0/1666223807 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f724002d730 con 0x7f7240034850 2024-03-29T11:26:00.622 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.634+0000 7f723dffb640 1 -- 172.21.15.135:0/1666223807 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f7238001950 con 0x7f7240034850 2024-03-29T11:26:00.622 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.634+0000 7f723dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:00.622 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.634+0000 7f723dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:00.622 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.634+0000 7f723dffb640 10 monclient(hunting): dump: 2024-03-29T11:26:00.622 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:00.622 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:00.622 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:00.622 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:00.622 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:00.623 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:00.623 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:00.623 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:00.623 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:00.623 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:00.623 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.635+0000 7f723dffb640 1 -- 172.21.15.135:0/1666223807 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3259134112 0 0) 0x7f724002d730 con 0x7f7240034850 2024-03-29T11:26:00.623 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.635+0000 7f723dffb640 1 -- 172.21.15.135:0/1666223807 >> v1:172.21.15.135:6790/0 conn(0x7f72400327d0 legacy=0x7f7240031400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.623 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.635+0000 7f723dffb640 1 -- 172.21.15.135:0/1666223807 >> v1:172.21.15.194:6789/0 conn(0x7f724002a640 legacy=0x7f72400bff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.623 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.635+0000 7f723dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:00.623 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.635+0000 7f723dffb640 1 monclient: found mon.a 2024-03-29T11:26:00.623 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.635+0000 7f723dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:00.623 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.635+0000 7f723dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:26:00.623 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.635+0000 7f723dffb640 1 -- 172.21.15.135:0/1666223807 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72400a89a0 con 0x7f7240034850 2024-03-29T11:26:00.623 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.635+0000 7f723dffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:00.623 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.635+0000 7f723dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:00.623 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.635+0000 7f723dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:00.624 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.635+0000 7f723dffb640 1 -- 172.21.15.135:0/1666223807 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72380036b0 con 0x7f7240034850 2024-03-29T11:26:00.624 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.635+0000 7f723dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:00.624 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.635+0000 7f723f7fe640 4 set_mon_vals no callback set 2024-03-29T11:26:00.624 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.636+0000 7f723dffb640 1 -- 172.21.15.135:0/1666223807 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f72380039c0 con 0x7f7240034850 2024-03-29T11:26:00.624 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.636+0000 7f723dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:00.624 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.636+0000 7f723dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:00.624 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.636+0000 7f723dffb640 10 monclient: dump: 2024-03-29T11:26:00.624 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:00.624 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:00.624 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:00.624 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:00.624 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:00.624 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:00.624 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:00.624 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:00.625 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:00.625 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:00.627 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.637+0000 7f724e612640 5 monclient: authenticate success, global_id 4296 2024-03-29T11:26:00.627 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.637+0000 7f724e612640 10 monclient: _renew_subs 2024-03-29T11:26:00.627 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.637+0000 7f724e612640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:26:00.627 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.637+0000 7f724e612640 1 -- 172.21.15.135:0/1666223807 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72400a8b90 con 0x7f7240034850 2024-03-29T11:26:00.627 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.637+0000 7f724e612640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:00.627 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.637+0000 7f724e612640 10 monclient: _renew_subs 2024-03-29T11:26:00.627 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.637+0000 7f724e612640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:26:00.627 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.637+0000 7f724e612640 1 -- 172.21.15.135:0/1666223807 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f724016d1f0 con 0x7f7240034850 2024-03-29T11:26:00.627 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.637+0000 7f724e612640 1 librados: init done 2024-03-29T11:26:00.627 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.639+0000 7f722b7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:00.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.639+0000 7f722b7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:00.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.639+0000 7f722b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:26:00.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.639+0000 7f722b7fe640 1 -- 172.21.15.135:0/1666223807 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7224005080 con 0x7f7240034850 2024-03-29T11:26:00.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.640+0000 7f723dffb640 1 -- 172.21.15.135:0/1666223807 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f7238019990 con 0x7f7240034850 2024-03-29T11:26:00.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.641+0000 7f723dffb640 1 -- 172.21.15.135:0/1666223807 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f7238083650 con 0x7f7240034850 2024-03-29T11:26:00.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.641+0000 7f723dffb640 10 client.4296.objecter ms_dispatch 0x7f7248000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:00.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.641+0000 7f723dffb640 3 client.4296.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:00.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.641+0000 7f724d610640 10 monclient: get_auth_request con 0x7f722c06dc40 auth_method 0 2024-03-29T11:26:00.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.641+0000 7f723dffb640 3 client.4296.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:00.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.641+0000 7f723dffb640 20 client.4296.objecter dump_active .. 0 homeless 2024-03-29T11:26:00.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.641+0000 7f723dffb640 10 client.4296.objecter ms_handle_connect 0x7f722c06dc40 2024-03-29T11:26:00.636 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.650+0000 7f723dffb640 1 -- 172.21.15.135:0/1666223807 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f723804ef30 con 0x7f7240034850 2024-03-29T11:26:00.636 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.651+0000 7f723dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:00.636 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.651+0000 7f723dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:00.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.973+0000 7f722b7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:00.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.973+0000 7f722b7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:00.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.973+0000 7f722b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:26:00.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.973+0000 7f722b7fe640 1 -- 172.21.15.135:0/1666223807 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7224005330 con 0x7f7240034850 2024-03-29T11:26:00.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.974+0000 7f723dffb640 1 -- 172.21.15.135:0/1666223807 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f7238050e60 con 0x7f7240034850 2024-03-29T11:26:00.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.974+0000 7f723dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:00.961 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.974+0000 7f723dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:00.961 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:26:00.961 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:00.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.978+0000 7f722b7fe640 10 librados: watch_flush enter 2024-03-29T11:26:00.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.978+0000 7f722b7fe640 10 librados: watch_flush exit 2024-03-29T11:26:00.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.978+0000 7f722b7fe640 20 client.4296.objecter shutdown clearing up homeless session... 2024-03-29T11:26:00.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.978+0000 7f722b7fe640 10 client.4296.objecter successfully canceled tick 2024-03-29T11:26:00.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.979+0000 7f722b7fe640 1 -- 172.21.15.135:0/1666223807 >> v1:172.21.15.135:6816/3469875285 conn(0x7f722c06dc40 legacy=0x7f722c070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.979+0000 7f722b7fe640 10 monclient: shutdown 2024-03-29T11:26:00.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.979+0000 7f722b7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:00.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.979+0000 7f722b7fe640 1 -- 172.21.15.135:0/1666223807 >> v1:172.21.15.135:6789/0 conn(0x7f7240034850 legacy=0x7f72400ba0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.980+0000 7f722b7fe640 1 -- 172.21.15.135:0/1666223807 shutdown_connections 2024-03-29T11:26:00.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.980+0000 7f722b7fe640 1 -- 172.21.15.135:0/1666223807 >> 172.21.15.135:0/1666223807 conn(0x7f72400058f0 msgr2=0x7f7240037f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:00.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.984+0000 7f722b7fe640 1 -- 172.21.15.135:0/1666223807 shutdown_connections 2024-03-29T11:26:00.971 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.984+0000 7f722b7fe640 1 -- 172.21.15.135:0/1666223807 wait complete. 2024-03-29T11:26:00.971 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:00.984+0000 7f722b7fe640 1 librados: shutdown 2024-03-29T11:26:00.987 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:07.795 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:07.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.962+0000 7f45abfc8640 10 monclient: get_monmap_and_config 2024-03-29T11:26:07.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.962+0000 7f45abfc8640 10 monclient: build_initial_monmap 2024-03-29T11:26:07.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.962+0000 7f45abfc8640 10 monclient: monmap: 2024-03-29T11:26:07.948 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:26:07.948 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:07.948 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:26:07.962804+0000 2024-03-29T11:26:07.948 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:26:07.962804+0000 2024-03-29T11:26:07.948 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:07.948 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:07.948 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:26:07.948 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:26:07.949 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:26:07.949 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:07.949 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.963+0000 7f45abfc8640 1 Processor -- start 2024-03-29T11:26:07.950 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.963+0000 7f45abfc8640 1 -- start start 2024-03-29T11:26:07.950 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.963+0000 7f45abfc8640 10 monclient: init 2024-03-29T11:26:07.950 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.964+0000 7f45abfc8640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:07.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.964+0000 7f45abfc8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:07.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.964+0000 7f45abfc8640 10 monclient(hunting): picked mon.noname-c con 0x7f45a40a9ac0 addr v1:172.21.15.194:6789/0 2024-03-29T11:26:07.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.965+0000 7f45abfc8640 10 monclient(hunting): picked mon.noname-b con 0x7f45a413ee90 addr v1:172.21.15.135:6790/0 2024-03-29T11:26:07.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.965+0000 7f45abfc8640 10 monclient(hunting): picked mon.noname-a con 0x7f45a40af900 addr v1:172.21.15.135:6789/0 2024-03-29T11:26:07.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.965+0000 7f45abfc8640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a40afd10 con 0x7f45a40af900 2024-03-29T11:26:07.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.965+0000 7f45abfc8640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a40af6f0 con 0x7f45a40a9ac0 2024-03-29T11:26:07.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.965+0000 7f45abfc8640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a40aac80 con 0x7f45a413ee90 2024-03-29T11:26:07.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.965+0000 7f45abfc8640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:07.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.965+0000 7f45abfc8640 10 monclient(hunting): authenticate will time out at 875.333559s 2024-03-29T11:26:07.955 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.970+0000 7f45aa565640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f45a40af900 0x7f45a4149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:39411/0 (socket says 172.21.15.135:39411) 2024-03-29T11:26:07.955 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.970+0000 7f45aa565640 1 -- 172.21.15.135:0/642574128 learned_addr learned my addr 172.21.15.135:0/642574128 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:26:07.956 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.971+0000 7f45a8d62640 1 -- 172.21.15.135:0/642574128 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316473823 0 0) 0x7f45a40afd10 con 0x7f45a40af900 2024-03-29T11:26:07.963 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 10 monclient(hunting): _negotiate 2024-03-29T11:26:07.963 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:07.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:07.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:26:07.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 1 -- 172.21.15.135:0/642574128 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4590003520 con 0x7f45a40af900 2024-03-29T11:26:07.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 1 -- 172.21.15.135:0/642574128 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163813814 0 0) 0x7f45a40af6f0 con 0x7f45a40a9ac0 2024-03-29T11:26:07.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 10 monclient(hunting): _negotiate 2024-03-29T11:26:07.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:07.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:07.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 10 monclient(hunting): my global_id is 4294 2024-03-29T11:26:07.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 1 -- 172.21.15.135:0/642574128 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45a40afd10 con 0x7f45a40a9ac0 2024-03-29T11:26:07.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 1 -- 172.21.15.135:0/642574128 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1455457056 0 0) 0x7f4590003520 con 0x7f45a40af900 2024-03-29T11:26:07.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 1 -- 172.21.15.135:0/642574128 >> v1:172.21.15.135:6790/0 conn(0x7f45a413ee90 legacy=0x7f45a413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 1 -- 172.21.15.135:0/642574128 >> v1:172.21.15.194:6789/0 conn(0x7f45a40a9ac0 legacy=0x7f45a40adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 10 monclient: _finish_hunting 0 2024-03-29T11:26:07.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 1 monclient: found mon.noname-a 2024-03-29T11:26:07.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:07.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.135:6789/0 2024-03-29T11:26:07.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 1 -- 172.21.15.135:0/642574128 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a40aaf60 con 0x7f45a40af900 2024-03-29T11:26:07.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 10 monclient: _finish_auth 0 2024-03-29T11:26:07.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 10 monclient: _check_auth_tickets 2024-03-29T11:26:07.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:07.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 1 -- 172.21.15.135:0/642574128 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f459c001a10 con 0x7f45a40af900 2024-03-29T11:26:07.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:07.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:07.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 10 monclient: dump: 2024-03-29T11:26:07.965 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:07.965 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:07.965 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:07.966 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:07.966 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:07.966 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:07.966 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:07.966 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:07.966 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:07.966 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:07.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 1 -- 172.21.15.135:0/642574128 <== mon.0 v1:172.21.15.135:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f459c001c30 con 0x7f45a40af900 2024-03-29T11:26:07.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:07.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 1 -- 172.21.15.135:0/642574128 <== mon.0 v1:172.21.15.135:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f459c001fc0 con 0x7f45a40af900 2024-03-29T11:26:07.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:07.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:07.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.972+0000 7f45a8d62640 10 monclient: dump: 2024-03-29T11:26:07.966 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:07.967 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:07.967 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:07.967 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:07.967 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:07.967 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:07.967 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:07.967 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:07.967 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:07.967 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:07.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.973+0000 7f45abfc8640 5 monclient: authenticate success, global_id 4299 2024-03-29T11:26:07.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.973+0000 7f45abfc8640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:07.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.973+0000 7f45abfc8640 4 set_mon_vals no callback set 2024-03-29T11:26:07.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.974+0000 7f45abfc8640 10 monclient: shutdown 2024-03-29T11:26:07.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.974+0000 7f45abfc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:07.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.974+0000 7f45abfc8640 1 -- 172.21.15.135:0/642574128 >> v1:172.21.15.135:6789/0 conn(0x7f45a40af900 legacy=0x7f45a4149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.974+0000 7f45abfc8640 1 -- 172.21.15.135:0/642574128 shutdown_connections 2024-03-29T11:26:07.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.974+0000 7f45abfc8640 1 -- 172.21.15.135:0/642574128 >> 172.21.15.135:0/642574128 conn(0x7f45a40b35e0 msgr2=0x7f45a40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:07.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.976+0000 7f45abfc8640 1 -- 172.21.15.135:0/642574128 shutdown_connections 2024-03-29T11:26:07.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.976+0000 7f45abfc8640 1 -- 172.21.15.135:0/642574128 wait complete. 2024-03-29T11:26:07.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.977+0000 7f45abfc8640 10 monclient: build_initial_monmap 2024-03-29T11:26:07.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.977+0000 7f45abfc8640 10 monclient: monmap: 2024-03-29T11:26:07.968 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:26:07.968 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:07.968 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:26:07.968 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:26:07.968 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:07.968 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:07.968 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:26:07.969 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:26:07.969 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:26:07.969 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:07.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.978+0000 7f45abfc8640 1 librados: starting msgr at 2024-03-29T11:26:07.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.978+0000 7f45abfc8640 1 librados: starting objecter 2024-03-29T11:26:07.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.978+0000 7f45abfc8640 1 Processor -- start 2024-03-29T11:26:07.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.978+0000 7f45abfc8640 1 -- start start 2024-03-29T11:26:07.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.979+0000 7f45abfc8640 1 librados: setting wanted keys 2024-03-29T11:26:07.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.979+0000 7f45abfc8640 1 librados: calling monclient init 2024-03-29T11:26:07.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.979+0000 7f45abfc8640 10 monclient: init 2024-03-29T11:26:07.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.982+0000 7f45abfc8640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:07.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.982+0000 7f45abfc8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:07.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.983+0000 7f45abfc8640 10 monclient(hunting): picked mon.noname-c con 0x7f45a40a9ac0 addr v1:172.21.15.135:6790/0 2024-03-29T11:26:07.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.983+0000 7f45abfc8640 10 monclient(hunting): picked mon.noname-b con 0x7f45a40af900 addr v1:172.21.15.194:6789/0 2024-03-29T11:26:07.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.984+0000 7f45abfc8640 10 monclient(hunting): picked mon.noname-a con 0x7f45a413ee90 addr v1:172.21.15.135:6789/0 2024-03-29T11:26:07.972 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.986+0000 7f45a9d64640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f45a40a9ac0 0x7f45a41dbcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:52647/0 (socket says 172.21.15.135:52647) 2024-03-29T11:26:07.972 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.986+0000 7f45a9d64640 1 -- 172.21.15.135:0/3623991814 learned_addr learned my addr 172.21.15.135:0/3623991814 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:26:07.972 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.986+0000 7f459a7fc640 10 client.?.objecter ms_handle_connect 0x7f45a40a9ac0 2024-03-29T11:26:07.972 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.986+0000 7f459a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:07.972 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.986+0000 7f45abfc8640 1 -- 172.21.15.135:0/3623991814 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a41db810 con 0x7f45a413ee90 2024-03-29T11:26:07.972 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.986+0000 7f45abfc8640 1 -- 172.21.15.135:0/3623991814 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a41db510 con 0x7f45a40af900 2024-03-29T11:26:07.972 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.986+0000 7f45abfc8640 1 -- 172.21.15.135:0/3623991814 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a411c400 con 0x7f45a40a9ac0 2024-03-29T11:26:07.972 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.986+0000 7f45abfc8640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:07.972 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.986+0000 7f45abfc8640 10 monclient(hunting): authenticate will time out at 875.354545s 2024-03-29T11:26:07.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.986+0000 7f459a7fc640 1 -- 172.21.15.135:0/3623991814 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191714012 0 0) 0x7f45a411c400 con 0x7f45a40a9ac0 2024-03-29T11:26:07.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.987+0000 7f459a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:07.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.987+0000 7f459a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:07.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.987+0000 7f459a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:07.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.987+0000 7f459a7fc640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:26:07.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.987+0000 7f459a7fc640 1 -- 172.21.15.135:0/3623991814 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4588004eb0 con 0x7f45a40a9ac0 2024-03-29T11:26:07.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.987+0000 7f459a7fc640 1 -- 172.21.15.135:0/3623991814 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1800835864 0 0) 0x7f4588004eb0 con 0x7f45a40a9ac0 2024-03-29T11:26:07.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.987+0000 7f459a7fc640 1 -- 172.21.15.135:0/3623991814 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45a411c400 con 0x7f45a40a9ac0 2024-03-29T11:26:07.976 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.987+0000 7f459a7fc640 1 -- 172.21.15.135:0/3623991814 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f45a0001a90 con 0x7f45a40a9ac0 2024-03-29T11:26:07.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.988+0000 7f459a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:07.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.988+0000 7f459a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:07.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.988+0000 7f459a7fc640 10 monclient(hunting): dump: 2024-03-29T11:26:07.977 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:07.977 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:07.977 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:07.977 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:07.977 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:07.977 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:07.977 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:07.977 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:07.977 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:07.977 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:07.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.988+0000 7f459a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:07.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.988+0000 7f459a7fc640 1 -- 172.21.15.135:0/3623991814 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3563641349 0 0) 0x7f45a411c400 con 0x7f45a40a9ac0 2024-03-29T11:26:07.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.988+0000 7f459a7fc640 1 -- 172.21.15.135:0/3623991814 >> v1:172.21.15.194:6789/0 conn(0x7f45a40af900 legacy=0x7f45a41da6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.988+0000 7f459a7fc640 1 -- 172.21.15.135:0/3623991814 >> v1:172.21.15.135:6789/0 conn(0x7f45a413ee90 legacy=0x7f45a41dae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.988+0000 7f459a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:07.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.988+0000 7f459a7fc640 1 monclient: found mon.c 2024-03-29T11:26:07.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.988+0000 7f459a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:07.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.988+0000 7f459a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:26:07.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.988+0000 7f459a7fc640 1 -- 172.21.15.135:0/3623991814 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a411c6e0 con 0x7f45a40a9ac0 2024-03-29T11:26:07.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.988+0000 7f459a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:07.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.989+0000 7f459a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:07.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.989+0000 7f459a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:07.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.989+0000 7f45abfc8640 5 monclient: authenticate success, global_id 4283 2024-03-29T11:26:07.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.989+0000 7f45abfc8640 10 monclient: _renew_subs 2024-03-29T11:26:07.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.989+0000 7f459a7fc640 1 -- 172.21.15.135:0/3623991814 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45a0001bf0 con 0x7f45a40a9ac0 2024-03-29T11:26:07.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.989+0000 7f45abfc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:26:07.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.989+0000 7f45abfc8640 1 -- 172.21.15.135:0/3623991814 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45a406f7f0 con 0x7f45a40a9ac0 2024-03-29T11:26:07.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.989+0000 7f45abfc8640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:07.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.989+0000 7f459a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:07.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.989+0000 7f459a7fc640 1 -- 172.21.15.135:0/3623991814 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f45a0003aa0 con 0x7f45a40a9ac0 2024-03-29T11:26:07.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.989+0000 7f459a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:07.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.989+0000 7f459a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:07.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.989+0000 7f459a7fc640 10 monclient: dump: 2024-03-29T11:26:07.979 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:07.979 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:07.979 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:07.979 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:07.979 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:07.979 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:07.980 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:07.980 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:07.980 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:07.980 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:07.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.989+0000 7f45abfc8640 10 monclient: _renew_subs 2024-03-29T11:26:07.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.989+0000 7f45abfc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:26:07.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.989+0000 7f45abfc8640 1 -- 172.21.15.135:0/3623991814 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a4072500 con 0x7f45a40a9ac0 2024-03-29T11:26:07.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.990+0000 7f45abfc8640 1 librados: init done 2024-03-29T11:26:07.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.990+0000 7f459bfff640 4 set_mon_vals no callback set 2024-03-29T11:26:07.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.991+0000 7f45abfc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:07.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.991+0000 7f45abfc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:07.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.991+0000 7f45abfc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:26:07.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.991+0000 7f45abfc8640 1 -- 172.21.15.135:0/3623991814 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f458c005080 con 0x7f45a40a9ac0 2024-03-29T11:26:07.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.996+0000 7f459a7fc640 1 -- 172.21.15.135:0/3623991814 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f45a0003c80 con 0x7f45a40a9ac0 2024-03-29T11:26:07.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.997+0000 7f459a7fc640 1 -- 172.21.15.135:0/3623991814 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f45a0084c20 con 0x7f45a40a9ac0 2024-03-29T11:26:07.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.997+0000 7f459a7fc640 10 client.4283.objecter ms_dispatch 0x7f45a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:07.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.997+0000 7f459a7fc640 3 client.4283.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:07.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.997+0000 7f459a7fc640 3 client.4283.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:07.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.997+0000 7f459a7fc640 20 client.4283.objecter dump_active .. 0 homeless 2024-03-29T11:26:07.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.998+0000 7f45a9563640 10 monclient: get_auth_request con 0x7f458806db10 auth_method 0 2024-03-29T11:26:07.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:07.998+0000 7f459a7fc640 10 client.4283.objecter ms_handle_connect 0x7f458806db10 2024-03-29T11:26:07.987 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:08.002+0000 7f459a7fc640 1 -- 172.21.15.135:0/3623991814 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f45a0050330 con 0x7f45a40a9ac0 2024-03-29T11:26:07.987 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:08.002+0000 7f459a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:07.987 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:08.002+0000 7f459a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:08.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:08.328+0000 7f45abfc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:08.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:08.328+0000 7f45abfc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:08.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:08.328+0000 7f45abfc8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:26:08.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:08.328+0000 7f45abfc8640 1 -- 172.21.15.135:0/3623991814 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f458c005330 con 0x7f45a40a9ac0 2024-03-29T11:26:08.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:08.330+0000 7f459a7fc640 1 -- 172.21.15.135:0/3623991814 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f45a0052430 con 0x7f45a40a9ac0 2024-03-29T11:26:08.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:08.330+0000 7f459a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:08.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:08.330+0000 7f459a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:08.316 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:26:08.316 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:08.319 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:08.334+0000 7f45abfc8640 10 librados: watch_flush enter 2024-03-29T11:26:08.319 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:08.334+0000 7f45abfc8640 10 librados: watch_flush exit 2024-03-29T11:26:08.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:08.334+0000 7f45abfc8640 20 client.4283.objecter shutdown clearing up homeless session... 2024-03-29T11:26:08.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:08.334+0000 7f45abfc8640 10 client.4283.objecter successfully canceled tick 2024-03-29T11:26:08.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:08.334+0000 7f45abfc8640 1 -- 172.21.15.135:0/3623991814 >> v1:172.21.15.135:6816/3469875285 conn(0x7f458806db10 legacy=0x7f458806ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:08.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:08.334+0000 7f45abfc8640 10 monclient: shutdown 2024-03-29T11:26:08.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:08.334+0000 7f45abfc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:08.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:08.334+0000 7f45abfc8640 1 -- 172.21.15.135:0/3623991814 >> v1:172.21.15.135:6790/0 conn(0x7f45a40a9ac0 legacy=0x7f45a41dbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:08.323 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:08.335+0000 7f45abfc8640 1 -- 172.21.15.135:0/3623991814 shutdown_connections 2024-03-29T11:26:08.323 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:08.335+0000 7f45abfc8640 1 -- 172.21.15.135:0/3623991814 >> 172.21.15.135:0/3623991814 conn(0x7f45a40b35e0 msgr2=0x7f45a40b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:08.323 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:08.338+0000 7f45abfc8640 1 -- 172.21.15.135:0/3623991814 shutdown_connections 2024-03-29T11:26:08.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:08.340+0000 7f45abfc8640 1 -- 172.21.15.135:0/3623991814 wait complete. 2024-03-29T11:26:08.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:08.340+0000 7f45abfc8640 1 librados: shutdown 2024-03-29T11:26:08.345 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:15.152 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:15.303 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.318+0000 7f16323c8640 10 monclient: get_monmap_and_config 2024-03-29T11:26:15.304 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.318+0000 7f16323c8640 10 monclient: build_initial_monmap 2024-03-29T11:26:15.304 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.318+0000 7f16323c8640 10 monclient: monmap: 2024-03-29T11:26:15.304 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:26:15.304 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:15.304 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:26:15.318920+0000 2024-03-29T11:26:15.304 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:26:15.318920+0000 2024-03-29T11:26:15.305 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:15.305 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:15.305 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:26:15.305 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:26:15.305 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:26:15.305 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:15.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.321+0000 7f16323c8640 1 Processor -- start 2024-03-29T11:26:15.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.321+0000 7f16323c8640 1 -- start start 2024-03-29T11:26:15.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.321+0000 7f16323c8640 10 monclient: init 2024-03-29T11:26:15.314 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.322+0000 7f16323c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:15.314 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.322+0000 7f16323c8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:15.314 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.322+0000 7f16323c8640 10 monclient(hunting): picked mon.noname-a con 0x7f162c0b1f60 addr v1:172.21.15.135:6789/0 2024-03-29T11:26:15.314 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.322+0000 7f16323c8640 10 monclient(hunting): picked mon.noname-c con 0x7f162c071eb0 addr v1:172.21.15.194:6789/0 2024-03-29T11:26:15.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.322+0000 7f16323c8640 10 monclient(hunting): picked mon.noname-b con 0x7f162c143d10 addr v1:172.21.15.135:6790/0 2024-03-29T11:26:15.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.322+0000 7f16323c8640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162c06ef80 con 0x7f162c0b1f60 2024-03-29T11:26:15.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.322+0000 7f16323c8640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162c06f260 con 0x7f162c071eb0 2024-03-29T11:26:15.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.322+0000 7f16323c8640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162c0b40c0 con 0x7f162c143d10 2024-03-29T11:26:15.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.322+0000 7f16323c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:15.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.322+0000 7f16323c8640 10 monclient(hunting): authenticate will time out at 882.690241s 2024-03-29T11:26:15.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.326+0000 7f162b7fe640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f162c071eb0 0x7f162c146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:36713/0 (socket says 172.21.15.135:36713) 2024-03-29T11:26:15.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.326+0000 7f162b7fe640 1 -- 172.21.15.135:0/162809262 learned_addr learned my addr 172.21.15.135:0/162809262 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:26:15.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.326+0000 7f162affd640 1 -- 172.21.15.135:0/162809262 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3299144222 0 0) 0x7f162c06ef80 con 0x7f162c0b1f60 2024-03-29T11:26:15.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.326+0000 7f162affd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:15.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.326+0000 7f162affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:15.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.326+0000 7f162affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:15.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.326+0000 7f162affd640 10 monclient(hunting): my global_id is 4302 2024-03-29T11:26:15.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.326+0000 7f162affd640 1 -- 172.21.15.135:0/162809262 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1614003520 con 0x7f162c0b1f60 2024-03-29T11:26:15.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 1 -- 172.21.15.135:0/162809262 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4234287540 0 0) 0x7f162c06f260 con 0x7f162c071eb0 2024-03-29T11:26:15.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:15.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:15.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:15.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 10 monclient(hunting): my global_id is 4297 2024-03-29T11:26:15.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 1 -- 172.21.15.135:0/162809262 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f162c06ef80 con 0x7f162c071eb0 2024-03-29T11:26:15.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 1 -- 172.21.15.135:0/162809262 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3825286147 0 0) 0x7f162c0b40c0 con 0x7f162c143d10 2024-03-29T11:26:15.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:15.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:15.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:15.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:26:15.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 1 -- 172.21.15.135:0/162809262 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f162c06f260 con 0x7f162c143d10 2024-03-29T11:26:15.317 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 1 -- 172.21.15.135:0/162809262 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1727455752 0 0) 0x7f162c06f260 con 0x7f162c143d10 2024-03-29T11:26:15.317 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 1 -- 172.21.15.135:0/162809262 >> v1:172.21.15.194:6789/0 conn(0x7f162c071eb0 legacy=0x7f162c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.317 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 1 -- 172.21.15.135:0/162809262 >> v1:172.21.15.135:6789/0 conn(0x7f162c0b1f60 legacy=0x7f162c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.317 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:15.317 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 1 monclient: found mon.noname-b 2024-03-29T11:26:15.317 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:15.317 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:26:15.317 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 1 -- 172.21.15.135:0/162809262 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f162c155d70 con 0x7f162c143d10 2024-03-29T11:26:15.317 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 10 monclient: _finish_auth 0 2024-03-29T11:26:15.317 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:15.317 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:15.317 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 1 -- 172.21.15.135:0/162809262 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f161c001570 con 0x7f162c143d10 2024-03-29T11:26:15.317 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:15.318 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:15.318 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 10 monclient: dump: 2024-03-29T11:26:15.318 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:15.318 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:15.318 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:15.318 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:15.318 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:15.318 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:15.318 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:15.318 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:15.318 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:15.318 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:15.318 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 1 -- 172.21.15.135:0/162809262 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f161c001740 con 0x7f162c143d10 2024-03-29T11:26:15.318 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:15.319 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 1 -- 172.21.15.135:0/162809262 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f161c001ad0 con 0x7f162c143d10 2024-03-29T11:26:15.319 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:15.319 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:15.319 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.327+0000 7f162affd640 10 monclient: dump: 2024-03-29T11:26:15.319 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:15.319 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:15.319 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:15.319 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:15.319 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:15.319 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:15.319 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:15.319 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:15.319 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:15.319 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:15.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.330+0000 7f16323c8640 5 monclient: authenticate success, global_id 4286 2024-03-29T11:26:15.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.330+0000 7f16323c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:15.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.330+0000 7f16323c8640 4 set_mon_vals no callback set 2024-03-29T11:26:15.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.331+0000 7f16323c8640 10 monclient: shutdown 2024-03-29T11:26:15.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.331+0000 7f16323c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:15.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.331+0000 7f16323c8640 1 -- 172.21.15.135:0/162809262 >> v1:172.21.15.135:6790/0 conn(0x7f162c143d10 legacy=0x7f162c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.331+0000 7f16323c8640 1 -- 172.21.15.135:0/162809262 shutdown_connections 2024-03-29T11:26:15.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.331+0000 7f16323c8640 1 -- 172.21.15.135:0/162809262 >> 172.21.15.135:0/162809262 conn(0x7f162c066a00 msgr2=0x7f162c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:15.321 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.331+0000 7f16323c8640 1 -- 172.21.15.135:0/162809262 shutdown_connections 2024-03-29T11:26:15.321 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.332+0000 7f16323c8640 1 -- 172.21.15.135:0/162809262 wait complete. 2024-03-29T11:26:15.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.336+0000 7f16323c8640 10 monclient: build_initial_monmap 2024-03-29T11:26:15.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.336+0000 7f16323c8640 10 monclient: monmap: 2024-03-29T11:26:15.322 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:26:15.322 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:15.322 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:26:15.322 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:26:15.322 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:15.322 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:15.322 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:26:15.322 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:26:15.322 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:26:15.323 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:15.323 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.336+0000 7f16323c8640 1 librados: starting msgr at 2024-03-29T11:26:15.323 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.336+0000 7f16323c8640 1 librados: starting objecter 2024-03-29T11:26:15.323 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.336+0000 7f16323c8640 1 Processor -- start 2024-03-29T11:26:15.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.338+0000 7f16323c8640 1 -- start start 2024-03-29T11:26:15.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.338+0000 7f16323c8640 1 librados: setting wanted keys 2024-03-29T11:26:15.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.338+0000 7f16323c8640 1 librados: calling monclient init 2024-03-29T11:26:15.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.338+0000 7f16323c8640 10 monclient: init 2024-03-29T11:26:15.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.339+0000 7f16323c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:15.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.339+0000 7f16323c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:15.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.339+0000 7f16323c8640 10 monclient(hunting): picked mon.noname-b con 0x7f162c0b1f60 addr v1:172.21.15.194:6789/0 2024-03-29T11:26:15.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.339+0000 7f16323c8640 10 monclient(hunting): picked mon.noname-c con 0x7f162c143d10 addr v1:172.21.15.135:6790/0 2024-03-29T11:26:15.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.339+0000 7f16323c8640 10 monclient(hunting): picked mon.noname-a con 0x7f162c1458f0 addr v1:172.21.15.135:6789/0 2024-03-29T11:26:15.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.339+0000 7f16323c8640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162c145e10 con 0x7f162c1458f0 2024-03-29T11:26:15.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.339+0000 7f16323c8640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162c0ac550 con 0x7f162c0b1f60 2024-03-29T11:26:15.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.339+0000 7f16323c8640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f162c0ac830 con 0x7f162c143d10 2024-03-29T11:26:15.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.339+0000 7f16323c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:15.327 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.339+0000 7f16323c8640 10 monclient(hunting): authenticate will time out at 882.707896s 2024-03-29T11:26:15.329 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.344+0000 7f162bfff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f162c0b1f60 0x7f162c13ff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:36793/0 (socket says 172.21.15.135:36793) 2024-03-29T11:26:15.330 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.344+0000 7f162bfff640 1 -- 172.21.15.135:0/2773198264 learned_addr learned my addr 172.21.15.135:0/2773198264 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:26:15.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.345+0000 7f1628ff9640 10 client.?.objecter ms_handle_connect 0x7f162c0b1f60 2024-03-29T11:26:15.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.345+0000 7f1628ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:15.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.345+0000 7f1628ff9640 10 client.?.objecter ms_handle_connect 0x7f162c143d10 2024-03-29T11:26:15.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.345+0000 7f1628ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:15.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.345+0000 7f1628ff9640 1 -- 172.21.15.135:0/2773198264 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834574502 0 0) 0x7f162c0ac550 con 0x7f162c0b1f60 2024-03-29T11:26:15.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.345+0000 7f1628ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:26:15.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.345+0000 7f1628ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:15.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.345+0000 7f1628ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:15.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.345+0000 7f1628ff9640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:26:15.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.345+0000 7f1628ff9640 1 -- 172.21.15.135:0/2773198264 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1618004eb0 con 0x7f162c0b1f60 2024-03-29T11:26:15.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.345+0000 7f1628ff9640 1 -- 172.21.15.135:0/2773198264 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931211923 0 0) 0x7f162c0ac830 con 0x7f162c143d10 2024-03-29T11:26:15.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.346+0000 7f1628ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:26:15.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.346+0000 7f1628ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:15.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.346+0000 7f1628ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:15.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.346+0000 7f1628ff9640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:26:15.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.346+0000 7f1628ff9640 1 -- 172.21.15.135:0/2773198264 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f162c0ac550 con 0x7f162c143d10 2024-03-29T11:26:15.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.346+0000 7f1628ff9640 1 -- 172.21.15.135:0/2773198264 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1655998940 0 0) 0x7f1618004eb0 con 0x7f162c0b1f60 2024-03-29T11:26:15.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.346+0000 7f1628ff9640 1 -- 172.21.15.135:0/2773198264 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f162c0ac830 con 0x7f162c0b1f60 2024-03-29T11:26:15.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.346+0000 7f1628ff9640 1 -- 172.21.15.135:0/2773198264 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f16200019b0 con 0x7f162c0b1f60 2024-03-29T11:26:15.337 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.346+0000 7f1628ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:15.337 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.346+0000 7f1628ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:15.337 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.346+0000 7f1628ff9640 10 monclient(hunting): dump: 2024-03-29T11:26:15.337 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:15.337 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:15.337 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:15.337 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:15.337 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:15.337 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:15.337 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:15.337 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:15.337 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:15.337 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:15.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.346+0000 7f1628ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:15.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.346+0000 7f1628ff9640 1 -- 172.21.15.135:0/2773198264 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1714292684 0 0) 0x7f162c0ac550 con 0x7f162c143d10 2024-03-29T11:26:15.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.346+0000 7f1628ff9640 1 -- 172.21.15.135:0/2773198264 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1618004eb0 con 0x7f162c143d10 2024-03-29T11:26:15.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.346+0000 7f1628ff9640 1 -- 172.21.15.135:0/2773198264 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f1624001cb0 con 0x7f162c143d10 2024-03-29T11:26:15.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.346+0000 7f1628ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:15.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.346+0000 7f1628ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:15.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.346+0000 7f1628ff9640 10 monclient(hunting): dump: 2024-03-29T11:26:15.338 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:15.338 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:15.338 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:15.338 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:15.338 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:15.338 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:15.339 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:15.339 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:15.339 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:15.339 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:15.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.347+0000 7f1628ff9640 1 -- 172.21.15.135:0/2773198264 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 229392428 0 0) 0x7f162c0ac830 con 0x7f162c0b1f60 2024-03-29T11:26:15.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.347+0000 7f1628ff9640 1 -- 172.21.15.135:0/2773198264 >> v1:172.21.15.135:6790/0 conn(0x7f162c143d10 legacy=0x7f162c1462a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.347+0000 7f1628ff9640 1 -- 172.21.15.135:0/2773198264 >> v1:172.21.15.135:6789/0 conn(0x7f162c1458f0 legacy=0x7f162c1469b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.347+0000 7f1628ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:26:15.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.347+0000 7f1628ff9640 1 monclient: found mon.b 2024-03-29T11:26:15.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.347+0000 7f1628ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:15.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.347+0000 7f1628ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:26:15.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.347+0000 7f1628ff9640 1 -- 172.21.15.135:0/2773198264 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f162c0aa440 con 0x7f162c0b1f60 2024-03-29T11:26:15.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.347+0000 7f1628ff9640 10 monclient: _finish_auth 0 2024-03-29T11:26:15.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.347+0000 7f1628ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:26:15.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.348+0000 7f1628ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:15.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.348+0000 7f16323c8640 5 monclient: authenticate success, global_id 4300 2024-03-29T11:26:15.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.348+0000 7f16323c8640 10 monclient: _renew_subs 2024-03-29T11:26:15.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.348+0000 7f16323c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:26:15.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.348+0000 7f16323c8640 1 -- 172.21.15.135:0/2773198264 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f162c140690 con 0x7f162c0b1f60 2024-03-29T11:26:15.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.348+0000 7f16323c8640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:15.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.348+0000 7f16323c8640 10 monclient: _renew_subs 2024-03-29T11:26:15.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.348+0000 7f16323c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:26:15.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.348+0000 7f16323c8640 1 -- 172.21.15.135:0/2773198264 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f162c0ab430 con 0x7f162c0b1f60 2024-03-29T11:26:15.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.348+0000 7f16323c8640 1 librados: init done 2024-03-29T11:26:15.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.348+0000 7f1628ff9640 10 client.4300.objecter ms_handle_connect 0x7f162c1458f0 2024-03-29T11:26:15.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.348+0000 7f1628ff9640 10 client.4300.objecter resend_mon_ops 2024-03-29T11:26:15.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.348+0000 7f1628ff9640 1 -- 172.21.15.135:0/2773198264 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1620001c70 con 0x7f162c0b1f60 2024-03-29T11:26:15.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.348+0000 7f1628ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:15.341 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.349+0000 7f1628ff9640 1 -- 172.21.15.135:0/2773198264 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f1620003a60 con 0x7f162c0b1f60 2024-03-29T11:26:15.341 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.349+0000 7f1629ffb640 4 set_mon_vals no callback set 2024-03-29T11:26:15.341 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.350+0000 7f1628ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:15.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.351+0000 7f1628ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:15.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.351+0000 7f1628ff9640 10 monclient: dump: 2024-03-29T11:26:15.342 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:15.342 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:15.342 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:15.342 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:15.342 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:15.342 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:15.342 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:15.343 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:15.343 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:15.343 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:15.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.351+0000 7f1628ff9640 1 -- 172.21.15.135:0/2773198264 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f1620003bc0 con 0x7f162c0b1f60 2024-03-29T11:26:15.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.352+0000 7f1628ff9640 1 -- 172.21.15.135:0/2773198264 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f1620084aa0 con 0x7f162c0b1f60 2024-03-29T11:26:15.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.352+0000 7f1628ff9640 10 client.4300.objecter ms_dispatch 0x7f162c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:15.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.352+0000 7f1628ff9640 3 client.4300.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:15.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.352+0000 7f1628ff9640 3 client.4300.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:15.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.352+0000 7f162b7fe640 10 monclient: get_auth_request con 0x7f161806de90 auth_method 0 2024-03-29T11:26:15.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.352+0000 7f16323c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:15.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.352+0000 7f16323c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:15.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.352+0000 7f16323c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:26:15.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.352+0000 7f16323c8640 1 -- 172.21.15.135:0/2773198264 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f160c005080 con 0x7f162c0b1f60 2024-03-29T11:26:15.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.353+0000 7f1628ff9640 20 client.4300.objecter dump_active .. 0 homeless 2024-03-29T11:26:15.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.353+0000 7f1628ff9640 10 client.4300.objecter ms_handle_connect 0x7f161806de90 2024-03-29T11:26:15.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.363+0000 7f1628ff9640 1 -- 172.21.15.135:0/2773198264 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1620050380 con 0x7f162c0b1f60 2024-03-29T11:26:15.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.363+0000 7f1628ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:15.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.363+0000 7f1628ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:15.671 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.685+0000 7f16323c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:15.671 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.685+0000 7f16323c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:15.671 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.685+0000 7f16323c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:26:15.671 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.685+0000 7f16323c8640 1 -- 172.21.15.135:0/2773198264 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f160c005330 con 0x7f162c0b1f60 2024-03-29T11:26:15.674 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.689+0000 7f1628ff9640 1 -- 172.21.15.135:0/2773198264 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f16200522b0 con 0x7f162c0b1f60 2024-03-29T11:26:15.674 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.689+0000 7f1628ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:15.675 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.689+0000 7f1628ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:15.675 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:26:15.675 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:15.678 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.693+0000 7f16323c8640 10 librados: watch_flush enter 2024-03-29T11:26:15.680 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.694+0000 7f16323c8640 10 librados: watch_flush exit 2024-03-29T11:26:15.680 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.694+0000 7f16323c8640 20 client.4300.objecter shutdown clearing up homeless session... 2024-03-29T11:26:15.680 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.694+0000 7f16323c8640 10 client.4300.objecter successfully canceled tick 2024-03-29T11:26:15.680 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.694+0000 7f16323c8640 1 -- 172.21.15.135:0/2773198264 >> v1:172.21.15.135:6816/3469875285 conn(0x7f161806de90 legacy=0x7f1618070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.680 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.694+0000 7f16323c8640 10 monclient: shutdown 2024-03-29T11:26:15.680 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.694+0000 7f16323c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:15.680 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.694+0000 7f16323c8640 1 -- 172.21.15.135:0/2773198264 >> v1:172.21.15.194:6789/0 conn(0x7f162c0b1f60 legacy=0x7f162c13ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.680 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.694+0000 7f16323c8640 1 -- 172.21.15.135:0/2773198264 shutdown_connections 2024-03-29T11:26:15.680 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.694+0000 7f16323c8640 1 -- 172.21.15.135:0/2773198264 >> 172.21.15.135:0/2773198264 conn(0x7f162c066a00 msgr2=0x7f162c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:15.680 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.695+0000 7f16323c8640 1 -- 172.21.15.135:0/2773198264 shutdown_connections 2024-03-29T11:26:15.680 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.695+0000 7f16323c8640 1 -- 172.21.15.135:0/2773198264 wait complete. 2024-03-29T11:26:15.680 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:15.695+0000 7f16323c8640 1 librados: shutdown 2024-03-29T11:26:15.697 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:22.505 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:22.661 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.676+0000 7ff074b4d640 10 monclient: get_monmap_and_config 2024-03-29T11:26:22.661 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.676+0000 7ff074b4d640 10 monclient: build_initial_monmap 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.676+0000 7ff074b4d640 10 monclient: monmap: 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:26:22.676859+0000 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:26:22.676859+0000 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:26:22.662 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:22.663 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.677+0000 7ff074b4d640 1 Processor -- start 2024-03-29T11:26:22.663 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.677+0000 7ff074b4d640 1 -- start start 2024-03-29T11:26:22.663 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.677+0000 7ff074b4d640 10 monclient: init 2024-03-29T11:26:22.667 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.678+0000 7ff074b4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:22.667 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.678+0000 7ff074b4d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:22.667 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.678+0000 7ff074b4d640 10 monclient(hunting): picked mon.noname-c con 0x7ff0700b1f60 addr v1:172.21.15.194:6789/0 2024-03-29T11:26:22.667 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.681+0000 7ff06e59c640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7ff0700b1f60 0x7ff070143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:50657/0 (socket says 172.21.15.135:50657) 2024-03-29T11:26:22.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.682+0000 7ff074b4d640 10 monclient(hunting): picked mon.noname-b con 0x7ff070071eb0 addr v1:172.21.15.135:6790/0 2024-03-29T11:26:22.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.682+0000 7ff074b4d640 10 monclient(hunting): picked mon.noname-a con 0x7ff070143d10 addr v1:172.21.15.135:6789/0 2024-03-29T11:26:22.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.682+0000 7ff074b4d640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff07006ef80 con 0x7ff070143d10 2024-03-29T11:26:22.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.682+0000 7ff074b4d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff07006f260 con 0x7ff0700b1f60 2024-03-29T11:26:22.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.682+0000 7ff074b4d640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0700b40c0 con 0x7ff070071eb0 2024-03-29T11:26:22.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.682+0000 7ff074b4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:22.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.682+0000 7ff074b4d640 10 monclient(hunting): authenticate will time out at 890.050652s 2024-03-29T11:26:22.669 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.682+0000 7ff06e59c640 1 -- 172.21.15.135:0/1165211995 learned_addr learned my addr 172.21.15.135:0/1165211995 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:26:22.669 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.684+0000 7ff06dd9b640 1 -- 172.21.15.135:0/1165211995 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2067789530 0 0) 0x7ff0700b40c0 con 0x7ff070071eb0 2024-03-29T11:26:22.670 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.684+0000 7ff06dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:26:22.670 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.684+0000 7ff06dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:22.670 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.684+0000 7ff06dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:22.670 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.684+0000 7ff06dd9b640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:26:22.670 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.684+0000 7ff06dd9b640 1 -- 172.21.15.135:0/1165211995 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff05c003520 con 0x7ff070071eb0 2024-03-29T11:26:22.670 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.685+0000 7ff06dd9b640 1 -- 172.21.15.135:0/1165211995 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3137521027 0 0) 0x7ff05c003520 con 0x7ff070071eb0 2024-03-29T11:26:22.670 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.685+0000 7ff06dd9b640 1 -- 172.21.15.135:0/1165211995 >> v1:172.21.15.194:6789/0 conn(0x7ff0700b1f60 legacy=0x7ff070143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:22.670 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.685+0000 7ff06dd9b640 1 -- 172.21.15.135:0/1165211995 >> v1:172.21.15.135:6789/0 conn(0x7ff070143d10 legacy=0x7ff070146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:22.673 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.686+0000 7ff06dd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:26:22.673 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.686+0000 7ff06dd9b640 1 monclient: found mon.noname-b 2024-03-29T11:26:22.673 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.686+0000 7ff06dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:22.673 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.686+0000 7ff06dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:26:22.673 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.686+0000 7ff06dd9b640 1 -- 172.21.15.135:0/1165211995 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff070155d70 con 0x7ff070071eb0 2024-03-29T11:26:22.673 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.686+0000 7ff06dd9b640 10 monclient: _finish_auth 0 2024-03-29T11:26:22.673 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.686+0000 7ff06dd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:26:22.673 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.686+0000 7ff06dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:22.673 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.686+0000 7ff06dd9b640 1 -- 172.21.15.135:0/1165211995 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7ff058001690 con 0x7ff070071eb0 2024-03-29T11:26:22.673 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.686+0000 7ff06dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:22.674 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.686+0000 7ff06dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:22.674 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.686+0000 7ff06dd9b640 10 monclient: dump: 2024-03-29T11:26:22.674 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:22.674 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:22.674 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:22.674 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:22.674 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:22.674 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:22.674 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:22.674 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:22.674 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:22.674 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:22.674 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.686+0000 7ff06dd9b640 1 -- 172.21.15.135:0/1165211995 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0580018f0 con 0x7ff070071eb0 2024-03-29T11:26:22.674 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.686+0000 7ff06dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:22.675 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.686+0000 7ff06dd9b640 1 -- 172.21.15.135:0/1165211995 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7ff058001d40 con 0x7ff070071eb0 2024-03-29T11:26:22.675 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.686+0000 7ff06dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:22.675 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.686+0000 7ff06dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:22.675 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.686+0000 7ff06dd9b640 10 monclient: dump: 2024-03-29T11:26:22.675 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:22.675 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:22.675 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:22.675 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:22.675 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:22.675 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:22.675 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:22.675 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:22.675 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:22.675 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:22.675 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.686+0000 7ff074b4d640 5 monclient: authenticate success, global_id 4292 2024-03-29T11:26:22.676 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.686+0000 7ff074b4d640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:22.676 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.686+0000 7ff074b4d640 4 set_mon_vals no callback set 2024-03-29T11:26:22.676 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.687+0000 7ff074b4d640 10 monclient: shutdown 2024-03-29T11:26:22.676 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.687+0000 7ff074b4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:22.676 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.687+0000 7ff074b4d640 1 -- 172.21.15.135:0/1165211995 >> v1:172.21.15.135:6790/0 conn(0x7ff070071eb0 legacy=0x7ff070146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:22.677 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.689+0000 7ff074b4d640 1 -- 172.21.15.135:0/1165211995 shutdown_connections 2024-03-29T11:26:22.677 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.689+0000 7ff074b4d640 1 -- 172.21.15.135:0/1165211995 >> 172.21.15.135:0/1165211995 conn(0x7ff070066a00 msgr2=0x7ff070066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:22.677 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.692+0000 7ff074b4d640 1 -- 172.21.15.135:0/1165211995 shutdown_connections 2024-03-29T11:26:22.678 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.693+0000 7ff074b4d640 1 -- 172.21.15.135:0/1165211995 wait complete. 2024-03-29T11:26:22.680 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.694+0000 7ff074b4d640 10 monclient: build_initial_monmap 2024-03-29T11:26:22.680 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.694+0000 7ff074b4d640 10 monclient: monmap: 2024-03-29T11:26:22.680 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:26:22.681 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:22.681 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:26:22.681 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:26:22.681 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:22.681 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:22.681 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:26:22.681 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:26:22.681 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:26:22.681 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:22.681 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.695+0000 7ff074b4d640 1 librados: starting msgr at 2024-03-29T11:26:22.681 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.695+0000 7ff074b4d640 1 librados: starting objecter 2024-03-29T11:26:22.681 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.695+0000 7ff074b4d640 1 Processor -- start 2024-03-29T11:26:22.682 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.696+0000 7ff074b4d640 1 -- start start 2024-03-29T11:26:22.682 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.696+0000 7ff074b4d640 1 librados: setting wanted keys 2024-03-29T11:26:22.682 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.696+0000 7ff074b4d640 1 librados: calling monclient init 2024-03-29T11:26:22.682 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.696+0000 7ff074b4d640 10 monclient: init 2024-03-29T11:26:22.683 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.697+0000 7ff074b4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:22.683 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.697+0000 7ff074b4d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:22.683 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.697+0000 7ff074b4d640 10 monclient(hunting): picked mon.noname-a con 0x7ff0700b1f60 addr v1:172.21.15.135:6789/0 2024-03-29T11:26:22.683 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.697+0000 7ff074b4d640 10 monclient(hunting): picked mon.noname-c con 0x7ff070143d10 addr v1:172.21.15.135:6790/0 2024-03-29T11:26:22.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.700+0000 7ff074b4d640 10 monclient(hunting): picked mon.noname-b con 0x7ff070133750 addr v1:172.21.15.194:6789/0 2024-03-29T11:26:22.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.700+0000 7ff074b4d640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff070070650 con 0x7ff0700b1f60 2024-03-29T11:26:22.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.700+0000 7ff074b4d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff070142460 con 0x7ff070133750 2024-03-29T11:26:22.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.700+0000 7ff074b4d640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0701363c0 con 0x7ff070143d10 2024-03-29T11:26:22.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.700+0000 7ff074b4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:22.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.700+0000 7ff074b4d640 10 monclient(hunting): authenticate will time out at 890.068417s 2024-03-29T11:26:22.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.701+0000 7ff06e59c640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7ff0700b1f60 0x7ff070134280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:43305/0 (socket says 172.21.15.135:43305) 2024-03-29T11:26:22.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.701+0000 7ff06e59c640 1 -- 172.21.15.135:0/706560648 learned_addr learned my addr 172.21.15.135:0/706560648 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:26:22.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.703+0000 7ff0677fe640 10 client.?.objecter ms_handle_connect 0x7ff0700b1f60 2024-03-29T11:26:22.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.703+0000 7ff0677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:22.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.703+0000 7ff0677fe640 1 -- 172.21.15.135:0/706560648 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310213232 0 0) 0x7ff070070650 con 0x7ff0700b1f60 2024-03-29T11:26:22.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.703+0000 7ff0677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:22.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.703+0000 7ff0677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:22.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.703+0000 7ff0677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:22.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.703+0000 7ff0677fe640 10 monclient(hunting): my global_id is 4308 2024-03-29T11:26:22.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.703+0000 7ff0677fe640 1 -- 172.21.15.135:0/706560648 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff050004eb0 con 0x7ff0700b1f60 2024-03-29T11:26:22.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.706+0000 7ff0677fe640 1 -- 172.21.15.135:0/706560648 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2649996293 0 0) 0x7ff050004eb0 con 0x7ff0700b1f60 2024-03-29T11:26:22.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.706+0000 7ff0677fe640 1 -- 172.21.15.135:0/706560648 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff070070650 con 0x7ff0700b1f60 2024-03-29T11:26:22.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.706+0000 7ff0677fe640 1 -- 172.21.15.135:0/706560648 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7ff060001510 con 0x7ff0700b1f60 2024-03-29T11:26:22.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.706+0000 7ff0677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:22.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.706+0000 7ff0677fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:22.694 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.706+0000 7ff0677fe640 10 monclient(hunting): dump: 2024-03-29T11:26:22.694 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:22.694 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:22.694 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:22.694 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:22.694 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:22.694 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:22.695 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:22.695 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:22.695 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:22.695 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:22.695 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.706+0000 7ff0677fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:22.695 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.706+0000 7ff0677fe640 10 client.?.objecter ms_handle_connect 0x7ff070133750 2024-03-29T11:26:22.695 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.706+0000 7ff0677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:22.695 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.706+0000 7ff0677fe640 1 -- 172.21.15.135:0/706560648 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927032350 0 0) 0x7ff070142460 con 0x7ff070133750 2024-03-29T11:26:22.695 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.706+0000 7ff0677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:22.695 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.706+0000 7ff0677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:22.695 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.706+0000 7ff0677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:22.695 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.706+0000 7ff0677fe640 10 monclient(hunting): my global_id is 4303 2024-03-29T11:26:22.695 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.706+0000 7ff0677fe640 1 -- 172.21.15.135:0/706560648 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff050004eb0 con 0x7ff070133750 2024-03-29T11:26:22.698 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.707+0000 7ff0677fe640 10 client.?.objecter ms_handle_connect 0x7ff070143d10 2024-03-29T11:26:22.698 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.707+0000 7ff0677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:22.698 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.707+0000 7ff0677fe640 1 -- 172.21.15.135:0/706560648 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4098854518 0 0) 0x7ff050004eb0 con 0x7ff070133750 2024-03-29T11:26:22.699 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.707+0000 7ff0677fe640 1 -- 172.21.15.135:0/706560648 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff070142460 con 0x7ff070133750 2024-03-29T11:26:22.699 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.707+0000 7ff0677fe640 1 -- 172.21.15.135:0/706560648 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7ff0680019d0 con 0x7ff070133750 2024-03-29T11:26:22.699 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.707+0000 7ff0677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:22.699 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.707+0000 7ff0677fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:22.699 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.707+0000 7ff0677fe640 10 monclient(hunting): dump: 2024-03-29T11:26:22.699 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:22.699 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:22.699 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:22.699 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:22.699 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:22.699 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:22.700 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:22.700 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:22.700 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:22.700 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:22.700 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.708+0000 7ff0677fe640 1 -- 172.21.15.135:0/706560648 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1302010589 0 0) 0x7ff070142460 con 0x7ff070133750 2024-03-29T11:26:22.700 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.708+0000 7ff0677fe640 1 -- 172.21.15.135:0/706560648 >> v1:172.21.15.135:6790/0 conn(0x7ff070143d10 legacy=0x7ff070142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:22.700 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.708+0000 7ff0677fe640 1 -- 172.21.15.135:0/706560648 >> v1:172.21.15.135:6789/0 conn(0x7ff0700b1f60 legacy=0x7ff070134280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:22.700 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.708+0000 7ff0677fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:22.700 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.708+0000 7ff0677fe640 1 monclient: found mon.b 2024-03-29T11:26:22.700 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.708+0000 7ff0677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:22.700 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.708+0000 7ff0677fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:26:22.700 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.708+0000 7ff0677fe640 1 -- 172.21.15.135:0/706560648 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff070142f90 con 0x7ff070133750 2024-03-29T11:26:22.700 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.708+0000 7ff0677fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:22.700 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.708+0000 7ff0677fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:22.701 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.708+0000 7ff0677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:22.701 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.709+0000 7ff0677fe640 1 -- 172.21.15.135:0/706560648 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff068001ee0 con 0x7ff070133750 2024-03-29T11:26:22.701 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.709+0000 7ff0677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:22.701 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.709+0000 7ff0677fe640 1 -- 172.21.15.135:0/706560648 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7ff068003b10 con 0x7ff070133750 2024-03-29T11:26:22.701 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.709+0000 7ff0677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:22.701 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.709+0000 7ff0677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:22.701 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.709+0000 7ff0677fe640 10 monclient: dump: 2024-03-29T11:26:22.701 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:22.701 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:22.701 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:22.701 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:22.701 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:22.701 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:22.701 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:22.702 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:22.702 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:22.702 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:22.702 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.709+0000 7ff06d59a640 4 set_mon_vals no callback set 2024-03-29T11:26:22.702 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.710+0000 7ff074b4d640 5 monclient: authenticate success, global_id 4303 2024-03-29T11:26:22.702 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.710+0000 7ff074b4d640 10 monclient: _renew_subs 2024-03-29T11:26:22.702 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.710+0000 7ff074b4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:26:22.702 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.710+0000 7ff074b4d640 1 -- 172.21.15.135:0/706560648 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff070070ae0 con 0x7ff070133750 2024-03-29T11:26:22.702 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.710+0000 7ff074b4d640 10 client.4303.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:22.702 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.710+0000 7ff074b4d640 10 monclient: _renew_subs 2024-03-29T11:26:22.702 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.710+0000 7ff074b4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:26:22.702 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.711+0000 7ff074b4d640 1 -- 172.21.15.135:0/706560648 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff070141270 con 0x7ff070133750 2024-03-29T11:26:22.702 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.711+0000 7ff074b4d640 1 librados: init done 2024-03-29T11:26:22.703 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.714+0000 7ff0677fe640 1 -- 172.21.15.135:0/706560648 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7ff068019ae0 con 0x7ff070133750 2024-03-29T11:26:22.704 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.714+0000 7ff074b4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:22.704 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.714+0000 7ff074b4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:22.704 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.714+0000 7ff074b4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:26:22.704 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.714+0000 7ff074b4d640 1 -- 172.21.15.135:0/706560648 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff070070800 con 0x7ff070133750 2024-03-29T11:26:22.704 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.715+0000 7ff0677fe640 1 -- 172.21.15.135:0/706560648 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7ff068083a10 con 0x7ff070133750 2024-03-29T11:26:22.704 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.715+0000 7ff0677fe640 10 client.4303.objecter ms_dispatch 0x7ff070000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:22.704 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.715+0000 7ff0677fe640 3 client.4303.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:22.704 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.715+0000 7ff0677fe640 3 client.4303.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:22.704 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.715+0000 7ff0677fe640 20 client.4303.objecter dump_active .. 0 homeless 2024-03-29T11:26:22.704 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.715+0000 7ff06e59c640 10 monclient: get_auth_request con 0x7ff05006df40 auth_method 0 2024-03-29T11:26:22.704 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.716+0000 7ff0677fe640 10 client.4303.objecter ms_handle_connect 0x7ff05006df40 2024-03-29T11:26:22.710 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.725+0000 7ff0677fe640 1 -- 172.21.15.135:0/706560648 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff06804f2f0 con 0x7ff070133750 2024-03-29T11:26:22.711 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.725+0000 7ff0677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:22.711 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:22.725+0000 7ff0677fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:23.034 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:23.047+0000 7ff074b4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:23.034 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:23.047+0000 7ff074b4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:23.034 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:23.047+0000 7ff074b4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:26:23.034 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:23.047+0000 7ff074b4d640 1 -- 172.21.15.135:0/706560648 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff070070c70 con 0x7ff070133750 2024-03-29T11:26:23.034 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:23.049+0000 7ff0677fe640 1 -- 172.21.15.135:0/706560648 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7ff068051220 con 0x7ff070133750 2024-03-29T11:26:23.035 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:23.049+0000 7ff0677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:23.035 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:23.049+0000 7ff0677fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:23.036 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:26:23.036 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:23.039 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:23.053+0000 7ff074b4d640 10 librados: watch_flush enter 2024-03-29T11:26:23.042 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:23.056+0000 7ff074b4d640 10 librados: watch_flush exit 2024-03-29T11:26:23.043 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:23.056+0000 7ff074b4d640 20 client.4303.objecter shutdown clearing up homeless session... 2024-03-29T11:26:23.043 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:23.056+0000 7ff074b4d640 10 client.4303.objecter successfully canceled tick 2024-03-29T11:26:23.043 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:23.056+0000 7ff074b4d640 1 -- 172.21.15.135:0/706560648 >> v1:172.21.15.135:6816/3469875285 conn(0x7ff05006df40 legacy=0x7ff050070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:23.043 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:23.057+0000 7ff074b4d640 10 monclient: shutdown 2024-03-29T11:26:23.043 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:23.057+0000 7ff074b4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:23.043 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:23.057+0000 7ff074b4d640 1 -- 172.21.15.135:0/706560648 >> v1:172.21.15.194:6789/0 conn(0x7ff070133750 legacy=0x7ff0701e5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:23.043 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:23.057+0000 7ff074b4d640 1 -- 172.21.15.135:0/706560648 shutdown_connections 2024-03-29T11:26:23.043 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:23.057+0000 7ff074b4d640 1 -- 172.21.15.135:0/706560648 >> 172.21.15.135:0/706560648 conn(0x7ff070066a00 msgr2=0x7ff0701468f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:23.043 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:23.057+0000 7ff074b4d640 1 -- 172.21.15.135:0/706560648 shutdown_connections 2024-03-29T11:26:23.043 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:23.057+0000 7ff074b4d640 1 -- 172.21.15.135:0/706560648 wait complete. 2024-03-29T11:26:23.043 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:23.057+0000 7ff074b4d640 1 librados: shutdown 2024-03-29T11:26:23.064 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:29.870 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:30.024 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.037+0000 7ff6743a4640 10 monclient: get_monmap_and_config 2024-03-29T11:26:30.024 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.037+0000 7ff6743a4640 10 monclient: build_initial_monmap 2024-03-29T11:26:30.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.037+0000 7ff6743a4640 10 monclient: monmap: 2024-03-29T11:26:30.025 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:26:30.025 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:30.025 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:26:30.038247+0000 2024-03-29T11:26:30.025 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:26:30.038247+0000 2024-03-29T11:26:30.025 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:30.025 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:30.025 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:26:30.025 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:26:30.025 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:26:30.025 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:30.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.038+0000 7ff6743a4640 1 Processor -- start 2024-03-29T11:26:30.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.039+0000 7ff6743a4640 1 -- start start 2024-03-29T11:26:30.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.039+0000 7ff6743a4640 10 monclient: init 2024-03-29T11:26:30.026 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.040+0000 7ff6743a4640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:30.026 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.040+0000 7ff6743a4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:30.026 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.040+0000 7ff6743a4640 10 monclient(hunting): picked mon.noname-b con 0x7ff66c0a9ac0 addr v1:172.21.15.135:6790/0 2024-03-29T11:26:30.026 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.040+0000 7ff6743a4640 10 monclient(hunting): picked mon.noname-c con 0x7ff66c13ee90 addr v1:172.21.15.194:6789/0 2024-03-29T11:26:30.027 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.041+0000 7ff6743a4640 10 monclient(hunting): picked mon.noname-a con 0x7ff66c0af900 addr v1:172.21.15.135:6789/0 2024-03-29T11:26:30.027 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.041+0000 7ff6743a4640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff66c0afd10 con 0x7ff66c0af900 2024-03-29T11:26:30.027 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.041+0000 7ff6743a4640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff66c0af6f0 con 0x7ff66c13ee90 2024-03-29T11:26:30.027 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.041+0000 7ff6743a4640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff66c0aac80 con 0x7ff66c0a9ac0 2024-03-29T11:26:30.028 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.041+0000 7ff6743a4640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:30.028 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.041+0000 7ff6743a4640 10 monclient(hunting): authenticate will time out at 897.409791s 2024-03-29T11:26:30.031 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.045+0000 7ff672941640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7ff66c0af900 0x7ff66c149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:35477/0 (socket says 172.21.15.135:35477) 2024-03-29T11:26:30.034 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.045+0000 7ff672941640 1 -- 172.21.15.135:0/1030452630 learned_addr learned my addr 172.21.15.135:0/1030452630 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:26:30.034 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 1 -- 172.21.15.135:0/1030452630 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1132519777 0 0) 0x7ff66c0afd10 con 0x7ff66c0af900 2024-03-29T11:26:30.034 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 10 monclient(hunting): _negotiate 2024-03-29T11:26:30.035 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:30.035 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:30.035 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:26:30.035 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 1 -- 172.21.15.135:0/1030452630 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff658003520 con 0x7ff66c0af900 2024-03-29T11:26:30.035 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 1 -- 172.21.15.135:0/1030452630 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3876706707 0 0) 0x7ff66c0af6f0 con 0x7ff66c13ee90 2024-03-29T11:26:30.036 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 10 monclient(hunting): _negotiate 2024-03-29T11:26:30.036 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:30.036 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:30.036 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 10 monclient(hunting): my global_id is 4306 2024-03-29T11:26:30.036 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 1 -- 172.21.15.135:0/1030452630 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff66c0afd10 con 0x7ff66c13ee90 2024-03-29T11:26:30.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 1 -- 172.21.15.135:0/1030452630 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 5267540 0 0) 0x7ff66c0afd10 con 0x7ff66c13ee90 2024-03-29T11:26:30.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 1 -- 172.21.15.135:0/1030452630 >> v1:172.21.15.135:6790/0 conn(0x7ff66c0a9ac0 legacy=0x7ff66c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:30.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 1 -- 172.21.15.135:0/1030452630 >> v1:172.21.15.135:6789/0 conn(0x7ff66c0af900 legacy=0x7ff66c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:30.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 10 monclient: _finish_hunting 0 2024-03-29T11:26:30.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 1 monclient: found mon.noname-c 2024-03-29T11:26:30.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:30.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:26:30.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 1 -- 172.21.15.135:0/1030452630 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff66c0aaf60 con 0x7ff66c13ee90 2024-03-29T11:26:30.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 10 monclient: _finish_auth 0 2024-03-29T11:26:30.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 10 monclient: _check_auth_tickets 2024-03-29T11:26:30.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:30.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff6743a4640 5 monclient: authenticate success, global_id 4306 2024-03-29T11:26:30.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff6743a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:30.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 1 -- 172.21.15.135:0/1030452630 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7ff6600015f0 con 0x7ff66c13ee90 2024-03-29T11:26:30.038 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:30.038 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:30.038 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 10 monclient: dump: 2024-03-29T11:26:30.038 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:30.038 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:30.038 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:30.038 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:30.038 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:30.038 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:30.038 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:30.038 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:30.038 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:30.038 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:30.038 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff6743a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:30.039 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 1 -- 172.21.15.135:0/1030452630 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6600017d0 con 0x7ff66c13ee90 2024-03-29T11:26:30.039 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:30.039 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 1 -- 172.21.15.135:0/1030452630 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7ff6600015f0 con 0x7ff66c13ee90 2024-03-29T11:26:30.039 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:30.039 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:30.039 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff67113e640 10 monclient: dump: 2024-03-29T11:26:30.039 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:30.039 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:30.039 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:30.039 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:30.039 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:30.039 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:30.039 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:30.039 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:30.040 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:30.040 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:30.040 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff6743a4640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:30.040 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.049+0000 7ff6743a4640 4 set_mon_vals no callback set 2024-03-29T11:26:30.040 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.050+0000 7ff6743a4640 10 monclient: shutdown 2024-03-29T11:26:30.040 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.050+0000 7ff6743a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:30.040 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.050+0000 7ff6743a4640 1 -- 172.21.15.135:0/1030452630 >> v1:172.21.15.194:6789/0 conn(0x7ff66c13ee90 legacy=0x7ff66c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:30.040 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.051+0000 7ff6743a4640 1 -- 172.21.15.135:0/1030452630 shutdown_connections 2024-03-29T11:26:30.040 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.051+0000 7ff6743a4640 1 -- 172.21.15.135:0/1030452630 >> 172.21.15.135:0/1030452630 conn(0x7ff66c0b35e0 msgr2=0x7ff66c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:30.040 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.054+0000 7ff6743a4640 1 -- 172.21.15.135:0/1030452630 shutdown_connections 2024-03-29T11:26:30.051 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.054+0000 7ff6743a4640 1 -- 172.21.15.135:0/1030452630 wait complete. 2024-03-29T11:26:30.051 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.055+0000 7ff6743a4640 10 monclient: build_initial_monmap 2024-03-29T11:26:30.051 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.055+0000 7ff6743a4640 10 monclient: monmap: 2024-03-29T11:26:30.051 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:26:30.051 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:30.051 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:26:30.051 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:26:30.051 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:30.051 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:30.051 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:26:30.051 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:26:30.051 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:26:30.052 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:30.052 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.055+0000 7ff6743a4640 1 librados: starting msgr at 2024-03-29T11:26:30.052 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.055+0000 7ff6743a4640 1 librados: starting objecter 2024-03-29T11:26:30.052 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.055+0000 7ff6743a4640 1 Processor -- start 2024-03-29T11:26:30.052 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.055+0000 7ff6743a4640 1 -- start start 2024-03-29T11:26:30.052 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.055+0000 7ff6743a4640 1 librados: setting wanted keys 2024-03-29T11:26:30.052 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.055+0000 7ff6743a4640 1 librados: calling monclient init 2024-03-29T11:26:30.052 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.055+0000 7ff6743a4640 10 monclient: init 2024-03-29T11:26:30.052 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.056+0000 7ff6743a4640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:30.052 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.056+0000 7ff6743a4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:30.052 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.060+0000 7ff6743a4640 10 monclient(hunting): picked mon.noname-c con 0x7ff66c0a9ac0 addr v1:172.21.15.135:6790/0 2024-03-29T11:26:30.052 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.060+0000 7ff6743a4640 10 monclient(hunting): picked mon.noname-b con 0x7ff66c0af900 addr v1:172.21.15.194:6789/0 2024-03-29T11:26:30.052 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.060+0000 7ff6743a4640 10 monclient(hunting): picked mon.noname-a con 0x7ff66c13ee90 addr v1:172.21.15.135:6789/0 2024-03-29T11:26:30.052 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.060+0000 7ff6743a4640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff66c11c4d0 con 0x7ff66c13ee90 2024-03-29T11:26:30.052 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.060+0000 7ff6743a4640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff66c11d070 con 0x7ff66c0af900 2024-03-29T11:26:30.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.060+0000 7ff6743a4640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff66c07de30 con 0x7ff66c0a9ac0 2024-03-29T11:26:30.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.060+0000 7ff6743a4640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:30.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.060+0000 7ff6743a4640 10 monclient(hunting): authenticate will time out at 897.428946s 2024-03-29T11:26:30.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.064+0000 7ff672941640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7ff66c13ee90 0x7ff66c07d720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:35631/0 (socket says 172.21.15.135:35631) 2024-03-29T11:26:30.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.064+0000 7ff672941640 1 -- 172.21.15.135:0/2595517696 learned_addr learned my addr 172.21.15.135:0/2595517696 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:26:30.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.064+0000 7ff65effd640 10 client.?.objecter ms_handle_connect 0x7ff66c13ee90 2024-03-29T11:26:30.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.064+0000 7ff65effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:30.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.064+0000 7ff65effd640 1 -- 172.21.15.135:0/2595517696 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3718922730 0 0) 0x7ff66c11c4d0 con 0x7ff66c13ee90 2024-03-29T11:26:30.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.064+0000 7ff65effd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:30.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.064+0000 7ff65effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:30.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.064+0000 7ff65effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:30.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.064+0000 7ff65effd640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:26:30.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.064+0000 7ff65effd640 1 -- 172.21.15.135:0/2595517696 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff650004eb0 con 0x7ff66c13ee90 2024-03-29T11:26:30.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.064+0000 7ff65effd640 1 -- 172.21.15.135:0/2595517696 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1378100743 0 0) 0x7ff650004eb0 con 0x7ff66c13ee90 2024-03-29T11:26:30.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.064+0000 7ff65effd640 1 -- 172.21.15.135:0/2595517696 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff66c11c4d0 con 0x7ff66c13ee90 2024-03-29T11:26:30.054 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.064+0000 7ff65effd640 1 -- 172.21.15.135:0/2595517696 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7ff664001840 con 0x7ff66c13ee90 2024-03-29T11:26:30.054 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.064+0000 7ff65effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:30.054 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.064+0000 7ff65effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:30.054 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.064+0000 7ff65effd640 10 monclient(hunting): dump: 2024-03-29T11:26:30.054 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:30.054 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:30.054 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:30.054 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:30.054 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:30.054 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:30.054 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:30.054 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:30.054 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:30.054 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:30.055 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.064+0000 7ff65effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:30.055 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.064+0000 7ff65effd640 10 client.?.objecter ms_handle_connect 0x7ff66c0a9ac0 2024-03-29T11:26:30.055 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.064+0000 7ff65effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:30.055 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.065+0000 7ff65effd640 1 -- 172.21.15.135:0/2595517696 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1985182797 0 0) 0x7ff66c11c4d0 con 0x7ff66c13ee90 2024-03-29T11:26:30.055 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.065+0000 7ff65effd640 1 -- 172.21.15.135:0/2595517696 >> v1:172.21.15.135:6790/0 conn(0x7ff66c0a9ac0 legacy=0x7ff66c11d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:30.055 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.065+0000 7ff65effd640 1 -- 172.21.15.135:0/2595517696 >> v1:172.21.15.194:6789/0 conn(0x7ff66c0af900 legacy=0x7ff66c07d010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:30.058 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.070+0000 7ff65effd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:30.058 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.070+0000 7ff65effd640 1 monclient: found mon.a 2024-03-29T11:26:30.058 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.070+0000 7ff65effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:30.058 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.070+0000 7ff65effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:26:30.058 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.070+0000 7ff65effd640 1 -- 172.21.15.135:0/2595517696 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff66c0725e0 con 0x7ff66c13ee90 2024-03-29T11:26:30.058 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.070+0000 7ff65effd640 10 monclient: _finish_auth 0 2024-03-29T11:26:30.058 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.070+0000 7ff65effd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:30.058 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.070+0000 7ff65effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:30.058 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.071+0000 7ff6743a4640 5 monclient: authenticate success, global_id 4314 2024-03-29T11:26:30.058 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.071+0000 7ff6743a4640 10 monclient: _renew_subs 2024-03-29T11:26:30.058 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.071+0000 7ff6743a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:26:30.058 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.071+0000 7ff6743a4640 1 -- 172.21.15.135:0/2595517696 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff66c11d2e0 con 0x7ff66c13ee90 2024-03-29T11:26:30.059 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.071+0000 7ff6743a4640 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:30.059 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.071+0000 7ff6743a4640 10 monclient: _renew_subs 2024-03-29T11:26:30.059 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.071+0000 7ff6743a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:26:30.059 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.071+0000 7ff6743a4640 1 -- 172.21.15.135:0/2595517696 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff66c11bac0 con 0x7ff66c13ee90 2024-03-29T11:26:30.059 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.071+0000 7ff6743a4640 1 librados: init done 2024-03-29T11:26:30.059 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.071+0000 7ff65effd640 1 -- 172.21.15.135:0/2595517696 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6640019a0 con 0x7ff66c13ee90 2024-03-29T11:26:30.059 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.071+0000 7ff65effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:30.059 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.071+0000 7ff65effd640 1 -- 172.21.15.135:0/2595517696 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7ff664004a40 con 0x7ff66c13ee90 2024-03-29T11:26:30.059 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.071+0000 7ff65effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:30.060 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.071+0000 7ff65effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:30.060 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.071+0000 7ff65effd640 10 monclient: dump: 2024-03-29T11:26:30.060 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:30.061 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:30.061 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:30.061 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:30.061 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:30.061 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:30.061 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:30.061 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:30.061 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:30.061 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:30.062 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.072+0000 7ff65ffff640 4 set_mon_vals no callback set 2024-03-29T11:26:30.063 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.073+0000 7ff65effd640 1 -- 172.21.15.135:0/2595517696 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7ff664004ba0 con 0x7ff66c13ee90 2024-03-29T11:26:30.063 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.074+0000 7ff6743a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:30.063 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.074+0000 7ff6743a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:30.063 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.074+0000 7ff6743a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:26:30.063 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.074+0000 7ff6743a4640 1 -- 172.21.15.135:0/2595517696 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff654005080 con 0x7ff66c13ee90 2024-03-29T11:26:30.063 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.074+0000 7ff65effd640 1 -- 172.21.15.135:0/2595517696 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7ff664084bb0 con 0x7ff66c13ee90 2024-03-29T11:26:30.063 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.074+0000 7ff65effd640 10 client.4314.objecter ms_dispatch 0x7ff66c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:30.063 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.074+0000 7ff65effd640 3 client.4314.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:30.063 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.074+0000 7ff65effd640 3 client.4314.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:30.063 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.074+0000 7ff65effd640 20 client.4314.objecter dump_active .. 0 homeless 2024-03-29T11:26:30.064 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.075+0000 7ff672140640 10 monclient: get_auth_request con 0x7ff65006dc10 auth_method 0 2024-03-29T11:26:30.064 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.075+0000 7ff65effd640 10 client.4314.objecter ms_handle_connect 0x7ff65006dc10 2024-03-29T11:26:30.068 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.083+0000 7ff65effd640 1 -- 172.21.15.135:0/2595517696 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff664050490 con 0x7ff66c13ee90 2024-03-29T11:26:30.069 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.083+0000 7ff65effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:30.069 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.083+0000 7ff65effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:30.389 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.403+0000 7ff6743a4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:30.389 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.403+0000 7ff6743a4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:30.390 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.403+0000 7ff6743a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:26:30.390 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.403+0000 7ff6743a4640 1 -- 172.21.15.135:0/2595517696 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff654005330 con 0x7ff66c13ee90 2024-03-29T11:26:30.390 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.404+0000 7ff65effd640 1 -- 172.21.15.135:0/2595517696 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7ff6640523c0 con 0x7ff66c13ee90 2024-03-29T11:26:30.390 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.404+0000 7ff65effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:30.390 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.404+0000 7ff65effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:30.391 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:26:30.391 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:30.397 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.407+0000 7ff6743a4640 10 librados: watch_flush enter 2024-03-29T11:26:30.398 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.412+0000 7ff6743a4640 10 librados: watch_flush exit 2024-03-29T11:26:30.398 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.412+0000 7ff6743a4640 20 client.4314.objecter shutdown clearing up homeless session... 2024-03-29T11:26:30.398 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.412+0000 7ff6743a4640 10 client.4314.objecter successfully canceled tick 2024-03-29T11:26:30.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.412+0000 7ff6743a4640 1 -- 172.21.15.135:0/2595517696 >> v1:172.21.15.135:6816/3469875285 conn(0x7ff65006dc10 legacy=0x7ff6500700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:30.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.412+0000 7ff6743a4640 10 monclient: shutdown 2024-03-29T11:26:30.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.412+0000 7ff6743a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:30.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.412+0000 7ff6743a4640 1 -- 172.21.15.135:0/2595517696 >> v1:172.21.15.135:6789/0 conn(0x7ff66c13ee90 legacy=0x7ff66c07d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:30.400 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.414+0000 7ff6743a4640 1 -- 172.21.15.135:0/2595517696 shutdown_connections 2024-03-29T11:26:30.400 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.414+0000 7ff6743a4640 1 -- 172.21.15.135:0/2595517696 >> 172.21.15.135:0/2595517696 conn(0x7ff66c0b35e0 msgr2=0x7ff66c12ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:30.401 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.414+0000 7ff6743a4640 1 -- 172.21.15.135:0/2595517696 shutdown_connections 2024-03-29T11:26:30.401 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.414+0000 7ff6743a4640 1 -- 172.21.15.135:0/2595517696 wait complete. 2024-03-29T11:26:30.401 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:30.415+0000 7ff6743a4640 1 librados: shutdown 2024-03-29T11:26:30.422 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:37.229 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:37.390 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.403+0000 7fea92895640 10 monclient: get_monmap_and_config 2024-03-29T11:26:37.391 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.403+0000 7fea92895640 10 monclient: build_initial_monmap 2024-03-29T11:26:37.391 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.403+0000 7fea92895640 10 monclient: monmap: 2024-03-29T11:26:37.391 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:26:37.391 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:37.391 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:26:37.405461+0000 2024-03-29T11:26:37.391 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:26:37.405461+0000 2024-03-29T11:26:37.391 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:37.391 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:37.391 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:26:37.391 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:26:37.391 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:26:37.391 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:37.392 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.405+0000 7fea92895640 1 Processor -- start 2024-03-29T11:26:37.392 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.405+0000 7fea92895640 1 -- start start 2024-03-29T11:26:37.392 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.405+0000 7fea92895640 10 monclient: init 2024-03-29T11:26:37.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.406+0000 7fea92895640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:37.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.406+0000 7fea92895640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:37.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.406+0000 7fea92895640 10 monclient(hunting): picked mon.noname-a con 0x7fea8c1422b0 addr v1:172.21.15.135:6789/0 2024-03-29T11:26:37.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.406+0000 7fea92895640 10 monclient(hunting): picked mon.noname-c con 0x7fea8c142690 addr v1:172.21.15.194:6789/0 2024-03-29T11:26:37.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.406+0000 7fea92895640 10 monclient(hunting): picked mon.noname-b con 0x7fea8c0b17d0 addr v1:172.21.15.135:6790/0 2024-03-29T11:26:37.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.406+0000 7fea92895640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea8c142c80 con 0x7fea8c1422b0 2024-03-29T11:26:37.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.406+0000 7fea92895640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea8c0aae40 con 0x7fea8c142690 2024-03-29T11:26:37.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.406+0000 7fea92895640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea8c13fbd0 con 0x7fea8c0b17d0 2024-03-29T11:26:37.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.406+0000 7fea92895640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:37.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.406+0000 7fea92895640 10 monclient(hunting): authenticate will time out at 904.775797s 2024-03-29T11:26:37.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.406+0000 7fea8bfff640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fea8c1422b0 0x7fea8c13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:44392/0 (socket says 172.21.15.135:44392) 2024-03-29T11:26:37.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.406+0000 7fea8bfff640 1 -- 172.21.15.135:0/1654885037 learned_addr learned my addr 172.21.15.135:0/1654885037 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:26:37.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.407+0000 7fea8affd640 1 -- 172.21.15.135:0/1654885037 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 255331373 0 0) 0x7fea8c0aae40 con 0x7fea8c142690 2024-03-29T11:26:37.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.407+0000 7fea8affd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:37.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.407+0000 7fea8affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:37.396 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.407+0000 7fea8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:37.397 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.407+0000 7fea8affd640 10 monclient(hunting): my global_id is 4309 2024-03-29T11:26:37.397 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.407+0000 7fea8affd640 1 -- 172.21.15.135:0/1654885037 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea74003520 con 0x7fea8c142690 2024-03-29T11:26:37.397 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.407+0000 7fea8affd640 1 -- 172.21.15.135:0/1654885037 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 956860113 0 0) 0x7fea74003520 con 0x7fea8c142690 2024-03-29T11:26:37.397 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.407+0000 7fea8affd640 1 -- 172.21.15.135:0/1654885037 >> v1:172.21.15.135:6790/0 conn(0x7fea8c0b17d0 legacy=0x7fea8c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:37.397 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.407+0000 7fea8affd640 1 -- 172.21.15.135:0/1654885037 >> v1:172.21.15.135:6789/0 conn(0x7fea8c1422b0 legacy=0x7fea8c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:37.397 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.408+0000 7fea8affd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:37.397 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.408+0000 7fea8affd640 1 monclient: found mon.noname-c 2024-03-29T11:26:37.397 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.408+0000 7fea8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:37.397 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.408+0000 7fea8affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:26:37.397 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.408+0000 7fea8affd640 1 -- 172.21.15.135:0/1654885037 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea8c151940 con 0x7fea8c142690 2024-03-29T11:26:37.397 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.408+0000 7fea8affd640 10 monclient: _finish_auth 0 2024-03-29T11:26:37.397 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.408+0000 7fea8affd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:37.397 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.408+0000 7fea8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:37.397 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.408+0000 7fea8affd640 1 -- 172.21.15.135:0/1654885037 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fea80001690 con 0x7fea8c142690 2024-03-29T11:26:37.398 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.408+0000 7fea8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:37.398 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.408+0000 7fea8affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:37.398 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.408+0000 7fea8affd640 10 monclient: dump: 2024-03-29T11:26:37.398 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:37.398 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:37.398 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:37.398 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:37.398 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:37.398 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:37.398 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:37.398 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:37.398 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:37.398 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:37.398 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.408+0000 7fea92895640 5 monclient: authenticate success, global_id 4309 2024-03-29T11:26:37.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.408+0000 7fea92895640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:37.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.408+0000 7fea8affd640 1 -- 172.21.15.135:0/1654885037 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea800017f0 con 0x7fea8c142690 2024-03-29T11:26:37.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.408+0000 7fea8affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:37.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.408+0000 7fea8affd640 1 -- 172.21.15.135:0/1654885037 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fea80001bc0 con 0x7fea8c142690 2024-03-29T11:26:37.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.408+0000 7fea8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:37.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.408+0000 7fea8affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:37.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.408+0000 7fea8affd640 10 monclient: dump: 2024-03-29T11:26:37.399 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:37.399 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:37.399 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:37.399 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:37.399 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:37.399 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:37.399 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:37.399 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:37.400 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:37.400 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:37.400 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.408+0000 7fea92895640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:37.400 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.408+0000 7fea92895640 4 set_mon_vals no callback set 2024-03-29T11:26:37.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.409+0000 7fea92895640 10 monclient: shutdown 2024-03-29T11:26:37.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.409+0000 7fea92895640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:37.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.409+0000 7fea92895640 1 -- 172.21.15.135:0/1654885037 >> v1:172.21.15.194:6789/0 conn(0x7fea8c142690 legacy=0x7fea8c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:37.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.409+0000 7fea92895640 1 -- 172.21.15.135:0/1654885037 shutdown_connections 2024-03-29T11:26:37.407 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.409+0000 7fea92895640 1 -- 172.21.15.135:0/1654885037 >> 172.21.15.135:0/1654885037 conn(0x7fea8c066a00 msgr2=0x7fea8c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:37.407 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.410+0000 7fea92895640 1 -- 172.21.15.135:0/1654885037 shutdown_connections 2024-03-29T11:26:37.407 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.410+0000 7fea92895640 1 -- 172.21.15.135:0/1654885037 wait complete. 2024-03-29T11:26:37.407 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.410+0000 7fea92895640 10 monclient: build_initial_monmap 2024-03-29T11:26:37.407 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.410+0000 7fea92895640 10 monclient: monmap: 2024-03-29T11:26:37.407 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:26:37.407 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:37.407 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:26:37.407 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:26:37.407 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:37.407 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:37.407 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:26:37.407 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:26:37.407 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:26:37.408 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:37.408 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.411+0000 7fea92895640 1 librados: starting msgr at 2024-03-29T11:26:37.408 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.411+0000 7fea92895640 1 librados: starting objecter 2024-03-29T11:26:37.408 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.411+0000 7fea92895640 1 Processor -- start 2024-03-29T11:26:37.408 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.412+0000 7fea92895640 1 -- start start 2024-03-29T11:26:37.408 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.412+0000 7fea92895640 1 librados: setting wanted keys 2024-03-29T11:26:37.408 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.412+0000 7fea92895640 1 librados: calling monclient init 2024-03-29T11:26:37.408 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.412+0000 7fea92895640 10 monclient: init 2024-03-29T11:26:37.408 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.413+0000 7fea92895640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:37.408 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.413+0000 7fea92895640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:37.408 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.413+0000 7fea92895640 10 monclient(hunting): picked mon.noname-a con 0x7fea8c0b17d0 addr v1:172.21.15.135:6789/0 2024-03-29T11:26:37.408 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.413+0000 7fea92895640 10 monclient(hunting): picked mon.noname-b con 0x7fea8c1422b0 addr v1:172.21.15.194:6789/0 2024-03-29T11:26:37.408 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.413+0000 7fea92895640 10 monclient(hunting): picked mon.noname-c con 0x7fea8c142690 addr v1:172.21.15.135:6790/0 2024-03-29T11:26:37.408 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.413+0000 7fea92895640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea8c0b6a90 con 0x7fea8c0b17d0 2024-03-29T11:26:37.408 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.413+0000 7fea92895640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea8c0ac7a0 con 0x7fea8c1422b0 2024-03-29T11:26:37.409 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.413+0000 7fea92895640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea8c0aca10 con 0x7fea8c142690 2024-03-29T11:26:37.409 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.413+0000 7fea92895640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:37.409 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.413+0000 7fea92895640 10 monclient(hunting): authenticate will time out at 904.782557s 2024-03-29T11:26:37.409 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.413+0000 7fea8bfff640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fea8c0b17d0 0x7fea8c0b54f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:45892/0 (socket says 172.21.15.135:45892) 2024-03-29T11:26:37.409 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.413+0000 7fea8bfff640 1 -- 172.21.15.135:0/620380603 learned_addr learned my addr 172.21.15.135:0/620380603 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:26:37.409 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.413+0000 7fea8b7fe640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fea8c1422b0 0x7fea8c0ab6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:41118/0 (socket says 172.21.15.135:41118) 2024-03-29T11:26:37.409 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.414+0000 7fea88ff9640 10 client.?.objecter ms_handle_connect 0x7fea8c0b17d0 2024-03-29T11:26:37.409 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.414+0000 7fea88ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:37.409 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.414+0000 7fea88ff9640 10 client.?.objecter ms_handle_connect 0x7fea8c1422b0 2024-03-29T11:26:37.409 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.414+0000 7fea88ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:37.409 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.414+0000 7fea88ff9640 10 client.?.objecter ms_handle_connect 0x7fea8c142690 2024-03-29T11:26:37.409 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.414+0000 7fea88ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:37.409 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.414+0000 7fea88ff9640 1 -- 172.21.15.135:0/620380603 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4244689693 0 0) 0x7fea8c0b6a90 con 0x7fea8c0b17d0 2024-03-29T11:26:37.409 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.414+0000 7fea88ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:26:37.409 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.414+0000 7fea88ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:37.410 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.414+0000 7fea88ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:37.410 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.414+0000 7fea88ff9640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:26:37.410 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.414+0000 7fea88ff9640 1 -- 172.21.15.135:0/620380603 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea78004eb0 con 0x7fea8c0b17d0 2024-03-29T11:26:37.410 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.414+0000 7fea88ff9640 1 -- 172.21.15.135:0/620380603 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053467150 0 0) 0x7fea8c0ac7a0 con 0x7fea8c1422b0 2024-03-29T11:26:37.410 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.414+0000 7fea88ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:26:37.410 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.414+0000 7fea88ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:37.410 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.414+0000 7fea88ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:37.410 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.414+0000 7fea88ff9640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:26:37.410 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.414+0000 7fea88ff9640 1 -- 172.21.15.135:0/620380603 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea8c0b6a90 con 0x7fea8c1422b0 2024-03-29T11:26:37.410 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.415+0000 7fea88ff9640 1 -- 172.21.15.135:0/620380603 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3205545149 0 0) 0x7fea8c0aca10 con 0x7fea8c142690 2024-03-29T11:26:37.410 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.415+0000 7fea88ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:26:37.410 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.415+0000 7fea88ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:37.410 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.415+0000 7fea88ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:37.410 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.415+0000 7fea88ff9640 10 monclient(hunting): my global_id is 4304 2024-03-29T11:26:37.410 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.415+0000 7fea88ff9640 1 -- 172.21.15.135:0/620380603 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea8c0ac7a0 con 0x7fea8c142690 2024-03-29T11:26:37.411 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.415+0000 7fea88ff9640 1 -- 172.21.15.135:0/620380603 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2457377513 0 0) 0x7fea8c0b6a90 con 0x7fea8c1422b0 2024-03-29T11:26:37.411 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.415+0000 7fea88ff9640 1 -- 172.21.15.135:0/620380603 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea8c0aca10 con 0x7fea8c1422b0 2024-03-29T11:26:37.411 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.415+0000 7fea88ff9640 1 -- 172.21.15.135:0/620380603 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fea80001b00 con 0x7fea8c1422b0 2024-03-29T11:26:37.411 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.415+0000 7fea88ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:37.411 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.415+0000 7fea88ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:37.412 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.415+0000 7fea88ff9640 10 monclient(hunting): dump: 2024-03-29T11:26:37.412 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:37.412 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:37.412 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:37.412 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:37.412 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:37.412 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:37.412 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:37.412 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:37.412 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:37.412 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:37.412 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.415+0000 7fea88ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:37.412 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.415+0000 7fea88ff9640 1 -- 172.21.15.135:0/620380603 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 589491383 0 0) 0x7fea78004eb0 con 0x7fea8c0b17d0 2024-03-29T11:26:37.412 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.415+0000 7fea88ff9640 1 -- 172.21.15.135:0/620380603 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea8c0b6a90 con 0x7fea8c0b17d0 2024-03-29T11:26:37.413 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.415+0000 7fea88ff9640 1 -- 172.21.15.135:0/620380603 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fea84001c70 con 0x7fea8c0b17d0 2024-03-29T11:26:37.413 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.415+0000 7fea88ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:37.413 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.415+0000 7fea88ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:37.413 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.415+0000 7fea88ff9640 10 monclient(hunting): dump: 2024-03-29T11:26:37.413 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:37.413 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:37.413 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:37.413 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:37.413 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:37.413 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:37.413 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:37.413 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:37.413 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:37.413 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:37.413 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.415+0000 7fea88ff9640 1 -- 172.21.15.135:0/620380603 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 63423438 0 0) 0x7fea8c0aca10 con 0x7fea8c1422b0 2024-03-29T11:26:37.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.416+0000 7fea88ff9640 1 -- 172.21.15.135:0/620380603 >> v1:172.21.15.135:6790/0 conn(0x7fea8c142690 legacy=0x7fea8c1e1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:37.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.416+0000 7fea88ff9640 1 -- 172.21.15.135:0/620380603 >> v1:172.21.15.135:6789/0 conn(0x7fea8c0b17d0 legacy=0x7fea8c0b54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:37.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.416+0000 7fea88ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:26:37.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.416+0000 7fea88ff9640 1 monclient: found mon.b 2024-03-29T11:26:37.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.416+0000 7fea88ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:37.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.416+0000 7fea88ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:26:37.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.416+0000 7fea88ff9640 1 -- 172.21.15.135:0/620380603 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea8c0accf0 con 0x7fea8c1422b0 2024-03-29T11:26:37.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.416+0000 7fea88ff9640 10 monclient: _finish_auth 0 2024-03-29T11:26:37.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.416+0000 7fea88ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:26:37.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.416+0000 7fea88ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:37.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.416+0000 7fea92895640 5 monclient: authenticate success, global_id 4312 2024-03-29T11:26:37.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.416+0000 7fea92895640 10 monclient: _renew_subs 2024-03-29T11:26:37.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.416+0000 7fea92895640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:26:37.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.416+0000 7fea92895640 1 -- 172.21.15.135:0/620380603 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea8c0b72e0 con 0x7fea8c1422b0 2024-03-29T11:26:37.415 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.416+0000 7fea92895640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:37.415 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.416+0000 7fea92895640 10 monclient: _renew_subs 2024-03-29T11:26:37.415 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.416+0000 7fea92895640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:26:37.415 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.416+0000 7fea92895640 1 -- 172.21.15.135:0/620380603 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea8c150710 con 0x7fea8c1422b0 2024-03-29T11:26:37.415 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.416+0000 7fea92895640 1 librados: init done 2024-03-29T11:26:37.415 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.416+0000 7fea88ff9640 1 -- 172.21.15.135:0/620380603 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea80001c60 con 0x7fea8c1422b0 2024-03-29T11:26:37.415 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.416+0000 7fea88ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:37.415 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.416+0000 7fea8a7fc640 4 set_mon_vals no callback set 2024-03-29T11:26:37.415 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.417+0000 7fea88ff9640 1 -- 172.21.15.135:0/620380603 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fea80003a40 con 0x7fea8c1422b0 2024-03-29T11:26:37.415 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.417+0000 7fea88ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:37.415 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.417+0000 7fea88ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:37.415 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.417+0000 7fea88ff9640 10 monclient: dump: 2024-03-29T11:26:37.415 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:37.415 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:37.415 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:37.416 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:37.416 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:37.416 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:37.416 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:37.416 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:37.416 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:37.416 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:37.416 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.419+0000 7fea88ff9640 1 -- 172.21.15.135:0/620380603 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fea80003ba0 con 0x7fea8c1422b0 2024-03-29T11:26:37.418 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.420+0000 7fea8bfff640 10 monclient: get_auth_request con 0x7fea7806df30 auth_method 0 2024-03-29T11:26:37.418 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.420+0000 7fea88ff9640 1 -- 172.21.15.135:0/620380603 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7fea800847f0 con 0x7fea8c1422b0 2024-03-29T11:26:37.418 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.420+0000 7fea88ff9640 10 client.4312.objecter ms_dispatch 0x7fea8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:37.419 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.420+0000 7fea88ff9640 3 client.4312.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:37.419 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.420+0000 7fea88ff9640 3 client.4312.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:37.419 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.420+0000 7fea88ff9640 20 client.4312.objecter dump_active .. 0 homeless 2024-03-29T11:26:37.419 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.420+0000 7fea88ff9640 10 client.4312.objecter ms_handle_connect 0x7fea7806df30 2024-03-29T11:26:37.419 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.421+0000 7fea92895640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:37.419 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.421+0000 7fea92895640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:37.419 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.421+0000 7fea92895640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:26:37.419 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.421+0000 7fea92895640 1 -- 172.21.15.135:0/620380603 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea8c0b6860 con 0x7fea8c1422b0 2024-03-29T11:26:37.419 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.429+0000 7fea88ff9640 1 -- 172.21.15.135:0/620380603 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea8c0b6860 con 0x7fea8c1422b0 2024-03-29T11:26:37.419 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.429+0000 7fea88ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:37.419 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.429+0000 7fea88ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:37.820 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.833+0000 7fea92895640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:37.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.833+0000 7fea92895640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:37.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.833+0000 7fea92895640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:26:37.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.833+0000 7fea92895640 1 -- 172.21.15.135:0/620380603 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea8c0b6860 con 0x7fea8c1422b0 2024-03-29T11:26:37.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.835+0000 7fea88ff9640 1 -- 172.21.15.135:0/620380603 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7fea8c0b6860 con 0x7fea8c1422b0 2024-03-29T11:26:37.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.835+0000 7fea88ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:37.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.835+0000 7fea88ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:37.822 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:26:37.822 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:37.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.839+0000 7fea92895640 10 librados: watch_flush enter 2024-03-29T11:26:37.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.839+0000 7fea92895640 10 librados: watch_flush exit 2024-03-29T11:26:37.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.839+0000 7fea92895640 20 client.4312.objecter shutdown clearing up homeless session... 2024-03-29T11:26:37.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.839+0000 7fea92895640 10 client.4312.objecter successfully canceled tick 2024-03-29T11:26:37.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.839+0000 7fea92895640 1 -- 172.21.15.135:0/620380603 >> v1:172.21.15.135:6816/3469875285 conn(0x7fea7806df30 legacy=0x7fea780703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:37.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.839+0000 7fea92895640 10 monclient: shutdown 2024-03-29T11:26:37.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.839+0000 7fea92895640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:37.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.839+0000 7fea92895640 1 -- 172.21.15.135:0/620380603 >> v1:172.21.15.194:6789/0 conn(0x7fea8c1422b0 legacy=0x7fea8c0ab6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:37.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.839+0000 7fea92895640 1 -- 172.21.15.135:0/620380603 shutdown_connections 2024-03-29T11:26:37.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.839+0000 7fea92895640 1 -- 172.21.15.135:0/620380603 >> 172.21.15.135:0/620380603 conn(0x7fea8c066a00 msgr2=0x7fea8c131bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:37.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.840+0000 7fea92895640 1 -- 172.21.15.135:0/620380603 shutdown_connections 2024-03-29T11:26:37.828 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.840+0000 7fea92895640 1 -- 172.21.15.135:0/620380603 wait complete. 2024-03-29T11:26:37.828 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:37.840+0000 7fea92895640 1 librados: shutdown 2024-03-29T11:26:37.846 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:44.654 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:44.804 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.816+0000 7f04d50ba640 10 monclient: get_monmap_and_config 2024-03-29T11:26:44.804 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.816+0000 7f04d50ba640 10 monclient: build_initial_monmap 2024-03-29T11:26:44.804 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.817+0000 7f04d50ba640 10 monclient: monmap: 2024-03-29T11:26:44.804 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:26:44.804 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:44.804 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:26:44.818600+0000 2024-03-29T11:26:44.804 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:26:44.818600+0000 2024-03-29T11:26:44.804 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:44.805 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:44.805 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:26:44.805 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:26:44.805 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:26:44.805 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:44.805 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.818+0000 7f04d50ba640 1 Processor -- start 2024-03-29T11:26:44.805 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.818+0000 7f04d50ba640 1 -- start start 2024-03-29T11:26:44.806 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.818+0000 7f04d50ba640 10 monclient: init 2024-03-29T11:26:44.809 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.819+0000 7f04d50ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:44.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.819+0000 7f04d50ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:44.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.819+0000 7f04d50ba640 10 monclient(hunting): picked mon.noname-a con 0x7f04d01457f0 addr v1:172.21.15.135:6789/0 2024-03-29T11:26:44.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.819+0000 7f04d50ba640 10 monclient(hunting): picked mon.noname-b con 0x7f04d013f650 addr v1:172.21.15.135:6790/0 2024-03-29T11:26:44.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.819+0000 7f04d50ba640 10 monclient(hunting): picked mon.noname-c con 0x7f04d00a9d70 addr v1:172.21.15.194:6789/0 2024-03-29T11:26:44.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.819+0000 7f04ced9d640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f04d01457f0 0x7f04d013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:57412/0 (socket says 172.21.15.135:57412) 2024-03-29T11:26:44.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.819+0000 7f04ced9d640 1 -- 172.21.15.135:0/4280332089 learned_addr learned my addr 172.21.15.135:0/4280332089 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:26:44.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.819+0000 7f04d50ba640 1 -- 172.21.15.135:0/4280332089 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d00aaef0 con 0x7f04d01457f0 2024-03-29T11:26:44.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.819+0000 7f04d50ba640 1 -- 172.21.15.135:0/4280332089 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d0145bd0 con 0x7f04d00a9d70 2024-03-29T11:26:44.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.820+0000 7f04d50ba640 1 -- 172.21.15.135:0/4280332089 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d0145eb0 con 0x7f04d013f650 2024-03-29T11:26:44.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.820+0000 7f04d50ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:44.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.820+0000 7f04d50ba640 10 monclient(hunting): authenticate will time out at 912.189246s 2024-03-29T11:26:44.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.820+0000 7f04cdd9b640 1 -- 172.21.15.135:0/4280332089 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4108730730 0 0) 0x7f04d0145eb0 con 0x7f04d013f650 2024-03-29T11:26:44.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.820+0000 7f04cdd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:26:44.811 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.820+0000 7f04cdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:44.811 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.820+0000 7f04cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:44.811 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.820+0000 7f04cdd9b640 10 monclient(hunting): my global_id is 4307 2024-03-29T11:26:44.811 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.820+0000 7f04cdd9b640 1 -- 172.21.15.135:0/4280332089 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b8003520 con 0x7f04d013f650 2024-03-29T11:26:44.811 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.820+0000 7f04cdd9b640 1 -- 172.21.15.135:0/4280332089 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 159248449 0 0) 0x7f04d00aaef0 con 0x7f04d01457f0 2024-03-29T11:26:44.811 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.820+0000 7f04cdd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:26:44.811 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.820+0000 7f04cdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:44.811 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.820+0000 7f04cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:44.811 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.820+0000 7f04cdd9b640 10 monclient(hunting): my global_id is 4323 2024-03-29T11:26:44.811 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.820+0000 7f04cdd9b640 1 -- 172.21.15.135:0/4280332089 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04d0145eb0 con 0x7f04d01457f0 2024-03-29T11:26:44.811 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.821+0000 7f04cdd9b640 1 -- 172.21.15.135:0/4280332089 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2187288415 0 0) 0x7f04d0145eb0 con 0x7f04d01457f0 2024-03-29T11:26:44.811 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.821+0000 7f04cdd9b640 1 -- 172.21.15.135:0/4280332089 >> v1:172.21.15.135:6790/0 conn(0x7f04d013f650 legacy=0x7f04d00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:44.811 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.821+0000 7f04cdd9b640 1 -- 172.21.15.135:0/4280332089 >> v1:172.21.15.194:6789/0 conn(0x7f04d00a9d70 legacy=0x7f04d014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:44.811 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.821+0000 7f04cdd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:26:44.812 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.821+0000 7f04cdd9b640 1 monclient: found mon.noname-a 2024-03-29T11:26:44.812 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.821+0000 7f04cdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:44.812 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.821+0000 7f04cdd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.135:6789/0 2024-03-29T11:26:44.812 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.821+0000 7f04cdd9b640 1 -- 172.21.15.135:0/4280332089 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d0146340 con 0x7f04d01457f0 2024-03-29T11:26:44.812 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.821+0000 7f04cdd9b640 10 monclient: _finish_auth 0 2024-03-29T11:26:44.812 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.821+0000 7f04cdd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:26:44.812 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.821+0000 7f04cdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:44.812 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.821+0000 7f04cdd9b640 1 -- 172.21.15.135:0/4280332089 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f04c00015a0 con 0x7f04d01457f0 2024-03-29T11:26:44.812 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.821+0000 7f04cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:44.812 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.821+0000 7f04cdd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:44.812 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.821+0000 7f04cdd9b640 10 monclient: dump: 2024-03-29T11:26:44.812 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:44.812 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:44.812 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:44.812 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:44.813 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:44.813 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:44.813 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:44.813 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:44.813 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:44.813 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:44.813 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.821+0000 7f04d50ba640 5 monclient: authenticate success, global_id 4323 2024-03-29T11:26:44.813 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.821+0000 7f04d50ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:44.813 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.821+0000 7f04cdd9b640 1 -- 172.21.15.135:0/4280332089 <== mon.0 v1:172.21.15.135:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04c00017c0 con 0x7f04d01457f0 2024-03-29T11:26:44.814 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.821+0000 7f04cdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:44.814 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.821+0000 7f04cdd9b640 1 -- 172.21.15.135:0/4280332089 <== mon.0 v1:172.21.15.135:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f04c0001b50 con 0x7f04d01457f0 2024-03-29T11:26:44.814 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.821+0000 7f04cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:44.814 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.821+0000 7f04cdd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:44.814 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.821+0000 7f04cdd9b640 10 monclient: dump: 2024-03-29T11:26:44.814 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:44.814 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:44.814 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:44.814 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:44.814 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:44.814 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:44.814 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:44.814 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:44.815 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:44.815 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:44.815 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.821+0000 7f04d50ba640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:44.815 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.821+0000 7f04d50ba640 4 set_mon_vals no callback set 2024-03-29T11:26:44.815 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.822+0000 7f04d50ba640 10 monclient: shutdown 2024-03-29T11:26:44.815 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.822+0000 7f04d50ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:44.815 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.822+0000 7f04d50ba640 1 -- 172.21.15.135:0/4280332089 >> v1:172.21.15.135:6789/0 conn(0x7f04d01457f0 legacy=0x7f04d013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:44.815 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.822+0000 7f04d50ba640 1 -- 172.21.15.135:0/4280332089 shutdown_connections 2024-03-29T11:26:44.815 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.822+0000 7f04d50ba640 1 -- 172.21.15.135:0/4280332089 >> 172.21.15.135:0/4280332089 conn(0x7f04d0066a00 msgr2=0x7f04d0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:44.815 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.822+0000 7f04d50ba640 1 -- 172.21.15.135:0/4280332089 shutdown_connections 2024-03-29T11:26:44.815 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.822+0000 7f04d50ba640 1 -- 172.21.15.135:0/4280332089 wait complete. 2024-03-29T11:26:44.820 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.824+0000 7f04d50ba640 10 monclient: build_initial_monmap 2024-03-29T11:26:44.820 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.824+0000 7f04d50ba640 10 monclient: monmap: 2024-03-29T11:26:44.821 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:26:44.821 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:44.821 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:26:44.821 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:26:44.821 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:44.821 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:44.821 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:26:44.821 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:26:44.821 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:26:44.821 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:44.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.824+0000 7f04d50ba640 1 librados: starting msgr at 2024-03-29T11:26:44.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.824+0000 7f04d50ba640 1 librados: starting objecter 2024-03-29T11:26:44.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.825+0000 7f04d50ba640 1 Processor -- start 2024-03-29T11:26:44.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.825+0000 7f04d50ba640 1 -- start start 2024-03-29T11:26:44.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.825+0000 7f04d50ba640 1 librados: setting wanted keys 2024-03-29T11:26:44.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.825+0000 7f04d50ba640 1 librados: calling monclient init 2024-03-29T11:26:44.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.825+0000 7f04d50ba640 10 monclient: init 2024-03-29T11:26:44.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.826+0000 7f04d50ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:44.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.826+0000 7f04d50ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:44.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.826+0000 7f04d50ba640 10 monclient(hunting): picked mon.noname-c con 0x7f04d00a9d70 addr v1:172.21.15.135:6790/0 2024-03-29T11:26:44.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.826+0000 7f04d50ba640 10 monclient(hunting): picked mon.noname-b con 0x7f04d013f650 addr v1:172.21.15.194:6789/0 2024-03-29T11:26:44.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.826+0000 7f04d50ba640 10 monclient(hunting): picked mon.noname-a con 0x7f04d01d7bf0 addr v1:172.21.15.135:6789/0 2024-03-29T11:26:44.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.826+0000 7f04d50ba640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d0142490 con 0x7f04d01d7bf0 2024-03-29T11:26:44.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.826+0000 7f04ce59c640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f04d013f650 0x7f04d0136b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:42452/0 (socket says 172.21.15.135:42452) 2024-03-29T11:26:44.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.826+0000 7f04ce59c640 1 -- 172.21.15.135:0/2026727797 learned_addr learned my addr 172.21.15.135:0/2026727797 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:26:44.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.826+0000 7f04d50ba640 1 -- 172.21.15.135:0/2026727797 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d00ac5d0 con 0x7f04d013f650 2024-03-29T11:26:44.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.826+0000 7f04d50ba640 1 -- 172.21.15.135:0/2026727797 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d01379b0 con 0x7f04d00a9d70 2024-03-29T11:26:44.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.826+0000 7f04d50ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:44.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.826+0000 7f04d50ba640 10 monclient(hunting): authenticate will time out at 912.195555s 2024-03-29T11:26:44.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.826+0000 7f04b77fe640 10 client.?.objecter ms_handle_connect 0x7f04d013f650 2024-03-29T11:26:44.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.826+0000 7f04b77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:44.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.827+0000 7f04b77fe640 10 client.?.objecter ms_handle_connect 0x7f04d01d7bf0 2024-03-29T11:26:44.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.827+0000 7f04b77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:44.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.827+0000 7f04b77fe640 10 client.?.objecter ms_handle_connect 0x7f04d00a9d70 2024-03-29T11:26:44.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.827+0000 7f04b77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:44.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.827+0000 7f04b77fe640 1 -- 172.21.15.135:0/2026727797 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3291336452 0 0) 0x7f04d00ac5d0 con 0x7f04d013f650 2024-03-29T11:26:44.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.827+0000 7f04b77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:44.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.827+0000 7f04b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:44.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.827+0000 7f04b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:44.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.827+0000 7f04b77fe640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:26:44.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.827+0000 7f04b77fe640 1 -- 172.21.15.135:0/2026727797 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04bc004eb0 con 0x7f04d013f650 2024-03-29T11:26:44.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.827+0000 7f04b77fe640 1 -- 172.21.15.135:0/2026727797 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676247289 0 0) 0x7f04d0142490 con 0x7f04d01d7bf0 2024-03-29T11:26:44.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.827+0000 7f04b77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:44.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.827+0000 7f04b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:44.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.827+0000 7f04b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:44.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.827+0000 7f04b77fe640 10 monclient(hunting): my global_id is 4326 2024-03-29T11:26:44.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.827+0000 7f04b77fe640 1 -- 172.21.15.135:0/2026727797 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04d00ac5d0 con 0x7f04d01d7bf0 2024-03-29T11:26:44.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.827+0000 7f04b77fe640 1 -- 172.21.15.135:0/2026727797 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4208227108 0 0) 0x7f04d01379b0 con 0x7f04d00a9d70 2024-03-29T11:26:44.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.827+0000 7f04b77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:44.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.827+0000 7f04b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:44.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.827+0000 7f04b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:44.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.827+0000 7f04b77fe640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:26:44.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.827+0000 7f04b77fe640 1 -- 172.21.15.135:0/2026727797 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04d0142490 con 0x7f04d00a9d70 2024-03-29T11:26:44.825 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.828+0000 7f04b77fe640 1 -- 172.21.15.135:0/2026727797 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1695408280 0 0) 0x7f04d00ac5d0 con 0x7f04d01d7bf0 2024-03-29T11:26:44.825 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.828+0000 7f04b77fe640 1 -- 172.21.15.135:0/2026727797 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04d01379b0 con 0x7f04d01d7bf0 2024-03-29T11:26:44.825 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.828+0000 7f04b77fe640 1 -- 172.21.15.135:0/2026727797 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f04c8001c30 con 0x7f04d01d7bf0 2024-03-29T11:26:44.825 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.828+0000 7f04b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:44.825 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.828+0000 7f04b77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:44.825 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.828+0000 7f04b77fe640 10 monclient(hunting): dump: 2024-03-29T11:26:44.825 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:44.825 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:44.825 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:44.825 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:44.825 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:44.825 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:44.825 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:44.825 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:44.825 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:44.826 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:44.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.828+0000 7f04b77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:44.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.828+0000 7f04b77fe640 1 -- 172.21.15.135:0/2026727797 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2293571522 0 0) 0x7f04bc004eb0 con 0x7f04d013f650 2024-03-29T11:26:44.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.828+0000 7f04b77fe640 1 -- 172.21.15.135:0/2026727797 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04d00ac5d0 con 0x7f04d013f650 2024-03-29T11:26:44.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.828+0000 7f04b77fe640 1 -- 172.21.15.135:0/2026727797 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f04c40018f0 con 0x7f04d013f650 2024-03-29T11:26:44.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.828+0000 7f04b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:44.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.828+0000 7f04b77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:44.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.828+0000 7f04b77fe640 10 monclient(hunting): dump: 2024-03-29T11:26:44.826 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:44.826 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:44.826 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:44.826 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:44.826 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:44.827 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:44.827 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:44.827 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:44.827 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:44.827 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:44.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.828+0000 7f04b77fe640 1 -- 172.21.15.135:0/2026727797 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 139846885 0 0) 0x7f04d0142490 con 0x7f04d00a9d70 2024-03-29T11:26:44.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.828+0000 7f04b77fe640 1 -- 172.21.15.135:0/2026727797 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04bc004eb0 con 0x7f04d00a9d70 2024-03-29T11:26:44.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.829+0000 7f04b77fe640 1 -- 172.21.15.135:0/2026727797 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f04c00014e0 con 0x7f04d00a9d70 2024-03-29T11:26:44.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.829+0000 7f04b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:44.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.829+0000 7f04b77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:44.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.829+0000 7f04b77fe640 10 monclient(hunting): dump: 2024-03-29T11:26:44.827 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:44.827 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:44.827 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:44.827 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.829+0000 7f04b77fe640 1 -- 172.21.15.135:0/2026727797 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2745404526 0 0) 0x7f04d01379b0 con 0x7f04d01d7bf0 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.829+0000 7f04b77fe640 1 -- 172.21.15.135:0/2026727797 >> v1:172.21.15.135:6790/0 conn(0x7f04d00a9d70 legacy=0x7f04d01d74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.829+0000 7f04b77fe640 1 -- 172.21.15.135:0/2026727797 >> v1:172.21.15.194:6789/0 conn(0x7f04d013f650 legacy=0x7f04d0136b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.829+0000 7f04b77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.829+0000 7f04b77fe640 1 monclient: found mon.a 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.829+0000 7f04b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:44.828 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.829+0000 7f04b77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:26:44.829 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.829+0000 7f04b77fe640 1 -- 172.21.15.135:0/2026727797 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d00ac820 con 0x7f04d01d7bf0 2024-03-29T11:26:44.829 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.829+0000 7f04b77fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:44.829 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.829+0000 7f04b77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:44.829 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.829+0000 7f04b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:44.829 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.829+0000 7f04d50ba640 5 monclient: authenticate success, global_id 4326 2024-03-29T11:26:44.829 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.829+0000 7f04d50ba640 10 monclient: _renew_subs 2024-03-29T11:26:44.829 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.829+0000 7f04d50ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:26:44.829 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.829+0000 7f04d50ba640 1 -- 172.21.15.135:0/2026727797 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04d0142120 con 0x7f04d01d7bf0 2024-03-29T11:26:44.829 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.829+0000 7f04d50ba640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:44.829 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.829+0000 7f04d50ba640 10 monclient: _renew_subs 2024-03-29T11:26:44.829 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.829+0000 7f04d50ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:26:44.829 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.830+0000 7f04d50ba640 1 -- 172.21.15.135:0/2026727797 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04d0142c80 con 0x7f04d01d7bf0 2024-03-29T11:26:44.829 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.830+0000 7f04d50ba640 1 librados: init done 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.830+0000 7f04b77fe640 1 -- 172.21.15.135:0/2026727797 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04c8003970 con 0x7f04d01d7bf0 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.830+0000 7f04b77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.830+0000 7f04cd59a640 4 set_mon_vals no callback set 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.831+0000 7f04b77fe640 1 -- 172.21.15.135:0/2026727797 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f04c8003cc0 con 0x7f04d01d7bf0 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.831+0000 7f04b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.831+0000 7f04b77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.831+0000 7f04b77fe640 10 monclient: dump: 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:44.830 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:44.831 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:44.831 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:44.831 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:44.831 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.831+0000 7f04d50ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:44.831 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.831+0000 7f04d50ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:44.831 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.831+0000 7f04d50ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:26:44.831 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.831+0000 7f04d50ba640 1 -- 172.21.15.135:0/2026727797 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b0005080 con 0x7f04d01d7bf0 2024-03-29T11:26:44.832 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.834+0000 7f04b77fe640 1 -- 172.21.15.135:0/2026727797 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f04c8003e20 con 0x7f04d01d7bf0 2024-03-29T11:26:44.832 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.835+0000 7f04b77fe640 1 -- 172.21.15.135:0/2026727797 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f04c8083a70 con 0x7f04d01d7bf0 2024-03-29T11:26:44.832 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.835+0000 7f04b77fe640 10 client.4326.objecter ms_dispatch 0x7f04d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:44.832 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.835+0000 7f04b77fe640 3 client.4326.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:44.832 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.835+0000 7f04b77fe640 3 client.4326.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:44.832 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.835+0000 7f04ced9d640 10 monclient: get_auth_request con 0x7f04bc06e0c0 auth_method 0 2024-03-29T11:26:44.832 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.835+0000 7f04b77fe640 20 client.4326.objecter dump_active .. 0 homeless 2024-03-29T11:26:44.832 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.836+0000 7f04b77fe640 10 client.4326.objecter ms_handle_connect 0x7f04bc06e0c0 2024-03-29T11:26:44.832 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.838+0000 7f04b77fe640 1 -- 172.21.15.135:0/2026727797 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f04c804f350 con 0x7f04d01d7bf0 2024-03-29T11:26:44.833 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.838+0000 7f04b77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:44.833 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:44.838+0000 7f04b77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:45.187 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:45.199+0000 7f04d50ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:45.187 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:45.200+0000 7f04d50ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:45.187 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:45.200+0000 7f04d50ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:26:45.187 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:45.200+0000 7f04d50ba640 1 -- 172.21.15.135:0/2026727797 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04b0005330 con 0x7f04d01d7bf0 2024-03-29T11:26:45.188 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:45.200+0000 7f04b77fe640 1 -- 172.21.15.135:0/2026727797 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f04c8051280 con 0x7f04d01d7bf0 2024-03-29T11:26:45.188 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:45.201+0000 7f04b77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:45.188 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:45.201+0000 7f04b77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:45.188 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:26:45.188 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:45.192 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:45.204+0000 7f04d50ba640 10 librados: watch_flush enter 2024-03-29T11:26:45.192 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:45.204+0000 7f04d50ba640 10 librados: watch_flush exit 2024-03-29T11:26:45.192 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:45.204+0000 7f04d50ba640 20 client.4326.objecter shutdown clearing up homeless session... 2024-03-29T11:26:45.192 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:45.204+0000 7f04d50ba640 10 client.4326.objecter successfully canceled tick 2024-03-29T11:26:45.192 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:45.204+0000 7f04d50ba640 1 -- 172.21.15.135:0/2026727797 >> v1:172.21.15.135:6816/3469875285 conn(0x7f04bc06e0c0 legacy=0x7f04bc070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:45.192 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:45.204+0000 7f04d50ba640 10 monclient: shutdown 2024-03-29T11:26:45.193 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:45.204+0000 7f04d50ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:45.193 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:45.204+0000 7f04d50ba640 1 -- 172.21.15.135:0/2026727797 >> v1:172.21.15.135:6789/0 conn(0x7f04d01d7bf0 legacy=0x7f04d01372a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:45.193 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:45.205+0000 7f04d50ba640 1 -- 172.21.15.135:0/2026727797 shutdown_connections 2024-03-29T11:26:45.193 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:45.205+0000 7f04d50ba640 1 -- 172.21.15.135:0/2026727797 >> 172.21.15.135:0/2026727797 conn(0x7f04d0066a00 msgr2=0x7f04d00b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:45.193 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:45.205+0000 7f04d50ba640 1 -- 172.21.15.135:0/2026727797 shutdown_connections 2024-03-29T11:26:45.193 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:45.205+0000 7f04d50ba640 1 -- 172.21.15.135:0/2026727797 wait complete. 2024-03-29T11:26:45.193 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:45.205+0000 7f04d50ba640 1 librados: shutdown 2024-03-29T11:26:45.207 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:52.015 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:52.161 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.174+0000 7febf21ca640 10 monclient: get_monmap_and_config 2024-03-29T11:26:52.161 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.174+0000 7febf21ca640 10 monclient: build_initial_monmap 2024-03-29T11:26:52.161 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.174+0000 7febf21ca640 10 monclient: monmap: 2024-03-29T11:26:52.162 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:26:52.162 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:52.162 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:26:52.174915+0000 2024-03-29T11:26:52.162 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:26:52.174915+0000 2024-03-29T11:26:52.162 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:52.162 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:52.163 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:26:52.163 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:26:52.163 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:26:52.163 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:52.165 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.176+0000 7febf21ca640 1 Processor -- start 2024-03-29T11:26:52.165 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.176+0000 7febf21ca640 1 -- start start 2024-03-29T11:26:52.166 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.176+0000 7febf21ca640 10 monclient: init 2024-03-29T11:26:52.166 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.177+0000 7febf21ca640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:52.166 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.177+0000 7febf21ca640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:52.166 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.177+0000 7febf21ca640 10 monclient(hunting): picked mon.noname-c con 0x7febec0b1f60 addr v1:172.21.15.194:6789/0 2024-03-29T11:26:52.166 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.177+0000 7febf21ca640 10 monclient(hunting): picked mon.noname-a con 0x7febec071eb0 addr v1:172.21.15.135:6789/0 2024-03-29T11:26:52.167 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.177+0000 7febf21ca640 10 monclient(hunting): picked mon.noname-b con 0x7febec143d10 addr v1:172.21.15.135:6790/0 2024-03-29T11:26:52.167 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.177+0000 7febf21ca640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febec06ef80 con 0x7febec071eb0 2024-03-29T11:26:52.167 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.177+0000 7febf21ca640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febec06f260 con 0x7febec0b1f60 2024-03-29T11:26:52.167 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.177+0000 7febf21ca640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febec0b40c0 con 0x7febec143d10 2024-03-29T11:26:52.167 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.177+0000 7febf21ca640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:52.167 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.177+0000 7febf21ca640 10 monclient(hunting): authenticate will time out at 919.545695s 2024-03-29T11:26:52.168 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.177+0000 7febebfff640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7febec143d10 0x7febec146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:60424/0 (socket says 172.21.15.135:60424) 2024-03-29T11:26:52.168 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.177+0000 7febebfff640 1 -- 172.21.15.135:0/1035957055 learned_addr learned my addr 172.21.15.135:0/1035957055 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:26:52.168 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.178+0000 7febea7fc640 1 -- 172.21.15.135:0/1035957055 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055426695 0 0) 0x7febec0b40c0 con 0x7febec143d10 2024-03-29T11:26:52.168 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.178+0000 7febea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:52.168 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.178+0000 7febea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:52.169 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.178+0000 7febea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:52.169 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.178+0000 7febea7fc640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:26:52.169 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.178+0000 7febea7fc640 1 -- 172.21.15.135:0/1035957055 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febd4003520 con 0x7febec143d10 2024-03-29T11:26:52.169 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.178+0000 7febea7fc640 1 -- 172.21.15.135:0/1035957055 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1261625670 0 0) 0x7febd4003520 con 0x7febec143d10 2024-03-29T11:26:52.169 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.178+0000 7febea7fc640 1 -- 172.21.15.135:0/1035957055 >> v1:172.21.15.194:6789/0 conn(0x7febec0b1f60 legacy=0x7febec143600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:52.173 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.181+0000 7febea7fc640 1 -- 172.21.15.135:0/1035957055 >> v1:172.21.15.135:6789/0 conn(0x7febec071eb0 legacy=0x7febec146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:52.173 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.183+0000 7febea7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:52.173 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.183+0000 7febea7fc640 1 monclient: found mon.noname-b 2024-03-29T11:26:52.173 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.183+0000 7febea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:52.173 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.183+0000 7febea7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:26:52.173 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.183+0000 7febea7fc640 1 -- 172.21.15.135:0/1035957055 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febec155d70 con 0x7febec143d10 2024-03-29T11:26:52.173 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.183+0000 7febea7fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:52.174 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.183+0000 7febea7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:52.174 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.183+0000 7febea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:52.174 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.183+0000 7febea7fc640 1 -- 172.21.15.135:0/1035957055 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7febe0001690 con 0x7febec143d10 2024-03-29T11:26:52.174 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.183+0000 7febea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:52.174 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.183+0000 7febea7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:52.174 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.183+0000 7febea7fc640 10 monclient: dump: 2024-03-29T11:26:52.174 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:52.174 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:52.175 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:52.175 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:52.175 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:52.175 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:52.175 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:52.175 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:52.175 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:52.175 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:52.176 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.183+0000 7febf21ca640 5 monclient: authenticate success, global_id 4313 2024-03-29T11:26:52.176 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.183+0000 7febf21ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:52.176 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.184+0000 7febea7fc640 1 -- 172.21.15.135:0/1035957055 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febe0001860 con 0x7febec143d10 2024-03-29T11:26:52.176 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.184+0000 7febea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:52.176 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.184+0000 7febea7fc640 1 -- 172.21.15.135:0/1035957055 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7febe0001cb0 con 0x7febec143d10 2024-03-29T11:26:52.176 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.184+0000 7febea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:52.177 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.184+0000 7febea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:52.177 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.184+0000 7febea7fc640 10 monclient: dump: 2024-03-29T11:26:52.177 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:52.178 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:52.178 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:52.178 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:52.178 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:52.178 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:52.178 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:52.179 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:52.179 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:52.179 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:52.179 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.184+0000 7febf21ca640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:52.180 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.184+0000 7febf21ca640 4 set_mon_vals no callback set 2024-03-29T11:26:52.180 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.185+0000 7febf21ca640 10 monclient: shutdown 2024-03-29T11:26:52.180 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.185+0000 7febf21ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:52.180 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.185+0000 7febf21ca640 1 -- 172.21.15.135:0/1035957055 >> v1:172.21.15.135:6790/0 conn(0x7febec143d10 legacy=0x7febec146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:52.180 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.186+0000 7febf21ca640 1 -- 172.21.15.135:0/1035957055 shutdown_connections 2024-03-29T11:26:52.180 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.186+0000 7febf21ca640 1 -- 172.21.15.135:0/1035957055 >> 172.21.15.135:0/1035957055 conn(0x7febec066a00 msgr2=0x7febec066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:52.180 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.186+0000 7febf21ca640 1 -- 172.21.15.135:0/1035957055 shutdown_connections 2024-03-29T11:26:52.189 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.187+0000 7febf21ca640 1 -- 172.21.15.135:0/1035957055 wait complete. 2024-03-29T11:26:52.189 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.188+0000 7febf21ca640 10 monclient: build_initial_monmap 2024-03-29T11:26:52.189 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.188+0000 7febf21ca640 10 monclient: monmap: 2024-03-29T11:26:52.189 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:26:52.189 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:52.190 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:26:52.190 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:26:52.191 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:52.191 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:52.191 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:26:52.191 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:26:52.191 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:26:52.191 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:52.192 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.188+0000 7febf21ca640 1 librados: starting msgr at 2024-03-29T11:26:52.192 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.188+0000 7febf21ca640 1 librados: starting objecter 2024-03-29T11:26:52.192 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.189+0000 7febf21ca640 1 Processor -- start 2024-03-29T11:26:52.192 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.189+0000 7febf21ca640 1 -- start start 2024-03-29T11:26:52.192 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.189+0000 7febf21ca640 1 librados: setting wanted keys 2024-03-29T11:26:52.192 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.189+0000 7febf21ca640 1 librados: calling monclient init 2024-03-29T11:26:52.193 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.189+0000 7febf21ca640 10 monclient: init 2024-03-29T11:26:52.193 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.190+0000 7febf21ca640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:52.193 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.190+0000 7febf21ca640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:52.193 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.190+0000 7febf21ca640 10 monclient(hunting): picked mon.noname-b con 0x7febec0b1f60 addr v1:172.21.15.194:6789/0 2024-03-29T11:26:52.193 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.190+0000 7febeaffd640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7febec143d10 0x7febec135d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:33692/0 (socket says 172.21.15.135:33692) 2024-03-29T11:26:52.193 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.190+0000 7febeaffd640 1 -- 172.21.15.135:0/2172897442 learned_addr learned my addr 172.21.15.135:0/2172897442 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:26:52.193 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.190+0000 7febd3fff640 10 client.?.objecter ms_handle_connect 0x7febec143d10 2024-03-29T11:26:52.194 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.190+0000 7febd3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:52.194 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.190+0000 7febf21ca640 10 monclient(hunting): picked mon.noname-c con 0x7febec143d10 addr v1:172.21.15.135:6790/0 2024-03-29T11:26:52.194 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.191+0000 7febd3fff640 10 client.?.objecter ms_handle_connect 0x7febec0b1f60 2024-03-29T11:26:52.194 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.191+0000 7febd3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:52.194 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.191+0000 7febf21ca640 10 monclient(hunting): picked mon.noname-a con 0x7febec13cce0 addr v1:172.21.15.135:6789/0 2024-03-29T11:26:52.194 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.191+0000 7febf21ca640 1 -- 172.21.15.135:0/2172897442 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febec0acc60 con 0x7febec13cce0 2024-03-29T11:26:52.194 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.191+0000 7febf21ca640 1 -- 172.21.15.135:0/2172897442 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febec0aceb0 con 0x7febec0b1f60 2024-03-29T11:26:52.195 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.191+0000 7febf21ca640 1 -- 172.21.15.135:0/2172897442 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febec0ab6e0 con 0x7febec143d10 2024-03-29T11:26:52.195 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.191+0000 7febf21ca640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:52.195 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.191+0000 7febf21ca640 10 monclient(hunting): authenticate will time out at 919.559962s 2024-03-29T11:26:52.195 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 1 -- 172.21.15.135:0/2172897442 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186106445 0 0) 0x7febec0aceb0 con 0x7febec0b1f60 2024-03-29T11:26:52.195 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:52.195 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:52.195 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:52.195 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:26:52.196 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 1 -- 172.21.15.135:0/2172897442 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febd8004eb0 con 0x7febec0b1f60 2024-03-29T11:26:52.196 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 1 -- 172.21.15.135:0/2172897442 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4118350807 0 0) 0x7febec0ab6e0 con 0x7febec143d10 2024-03-29T11:26:52.196 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:52.196 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:52.196 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:52.196 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:26:52.196 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 1 -- 172.21.15.135:0/2172897442 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febec0aceb0 con 0x7febec143d10 2024-03-29T11:26:52.196 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 1 -- 172.21.15.135:0/2172897442 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1872253358 0 0) 0x7febd8004eb0 con 0x7febec0b1f60 2024-03-29T11:26:52.197 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 1 -- 172.21.15.135:0/2172897442 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febec0ab6e0 con 0x7febec0b1f60 2024-03-29T11:26:52.197 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 1 -- 172.21.15.135:0/2172897442 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7febdc0017f0 con 0x7febec0b1f60 2024-03-29T11:26:52.197 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:52.197 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:52.197 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 10 monclient(hunting): dump: 2024-03-29T11:26:52.197 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:52.197 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:52.198 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:52.198 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:52.198 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:52.198 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:52.198 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:52.198 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:52.199 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:52.199 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:52.199 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:52.199 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 1 -- 172.21.15.135:0/2172897442 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3498527969 0 0) 0x7febec0aceb0 con 0x7febec143d10 2024-03-29T11:26:52.199 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 1 -- 172.21.15.135:0/2172897442 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febd8004eb0 con 0x7febec143d10 2024-03-29T11:26:52.199 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 1 -- 172.21.15.135:0/2172897442 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7febe4001ca0 con 0x7febec143d10 2024-03-29T11:26:52.200 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:52.200 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:52.200 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 10 monclient(hunting): dump: 2024-03-29T11:26:52.200 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:52.200 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:52.200 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:52.201 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:52.201 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:52.201 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:52.201 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:52.201 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:52.201 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:52.201 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:52.201 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 1 -- 172.21.15.135:0/2172897442 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1510827795 0 0) 0x7febec0ab6e0 con 0x7febec0b1f60 2024-03-29T11:26:52.201 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 1 -- 172.21.15.135:0/2172897442 >> v1:172.21.15.135:6790/0 conn(0x7febec143d10 legacy=0x7febec135d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:52.201 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.192+0000 7febd3fff640 1 -- 172.21.15.135:0/2172897442 >> v1:172.21.15.135:6789/0 conn(0x7febec13cce0 legacy=0x7febec136420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:52.201 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.193+0000 7febd3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:52.202 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.193+0000 7febd3fff640 1 monclient: found mon.b 2024-03-29T11:26:52.202 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.193+0000 7febd3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:52.202 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.193+0000 7febd3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:26:52.202 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.193+0000 7febd3fff640 1 -- 172.21.15.135:0/2172897442 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febec0aa850 con 0x7febec0b1f60 2024-03-29T11:26:52.202 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.193+0000 7febd3fff640 10 monclient: _finish_auth 0 2024-03-29T11:26:52.202 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.193+0000 7febd3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:52.202 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.193+0000 7febd3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:52.202 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.193+0000 7febf21ca640 5 monclient: authenticate success, global_id 4318 2024-03-29T11:26:52.202 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.193+0000 7febf21ca640 10 monclient: _renew_subs 2024-03-29T11:26:52.202 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.193+0000 7febf21ca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:26:52.202 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.193+0000 7febf21ca640 1 -- 172.21.15.135:0/2172897442 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febec0abc00 con 0x7febec0b1f60 2024-03-29T11:26:52.202 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.193+0000 7febf21ca640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:52.202 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.193+0000 7febf21ca640 10 monclient: _renew_subs 2024-03-29T11:26:52.203 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.193+0000 7febf21ca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:26:52.203 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.193+0000 7febf21ca640 1 -- 172.21.15.135:0/2172897442 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febec0706e0 con 0x7febec0b1f60 2024-03-29T11:26:52.203 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.193+0000 7febf21ca640 1 librados: init done 2024-03-29T11:26:52.203 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.193+0000 7febd3fff640 1 -- 172.21.15.135:0/2172897442 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febdc001d00 con 0x7febec0b1f60 2024-03-29T11:26:52.203 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.193+0000 7febd3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:52.203 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.193+0000 7febd3fff640 1 -- 172.21.15.135:0/2172897442 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7febdc003830 con 0x7febec0b1f60 2024-03-29T11:26:52.203 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.193+0000 7febd3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:52.203 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.193+0000 7febd3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:52.203 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.193+0000 7febd3fff640 10 monclient: dump: 2024-03-29T11:26:52.203 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:52.203 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:52.203 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:52.203 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:52.203 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:52.204 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:52.204 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:52.204 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:52.204 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:52.204 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:52.204 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.193+0000 7febe9ffb640 4 set_mon_vals no callback set 2024-03-29T11:26:52.204 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.196+0000 7febd3fff640 1 -- 172.21.15.135:0/2172897442 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7febdc019800 con 0x7febec0b1f60 2024-03-29T11:26:52.204 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.196+0000 7febf21ca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:52.204 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.196+0000 7febf21ca640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:52.204 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.196+0000 7febf21ca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:26:52.204 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.196+0000 7febf21ca640 1 -- 172.21.15.135:0/2172897442 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febcc005080 con 0x7febec0b1f60 2024-03-29T11:26:52.204 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.196+0000 7febd3fff640 1 -- 172.21.15.135:0/2172897442 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7febdc083830 con 0x7febec0b1f60 2024-03-29T11:26:52.204 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.196+0000 7febd3fff640 10 client.4318.objecter ms_dispatch 0x7febec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:52.204 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.196+0000 7febd3fff640 3 client.4318.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:52.204 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.196+0000 7febd3fff640 3 client.4318.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:52.205 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.196+0000 7febd3fff640 20 client.4318.objecter dump_active .. 0 homeless 2024-03-29T11:26:52.205 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.196+0000 7febeaffd640 10 monclient: get_auth_request con 0x7febd806dd60 auth_method 0 2024-03-29T11:26:52.205 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.197+0000 7febd3fff640 10 client.4318.objecter ms_handle_connect 0x7febd806dd60 2024-03-29T11:26:52.205 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.206+0000 7febd3fff640 1 -- 172.21.15.135:0/2172897442 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7febdc04f110 con 0x7febec0b1f60 2024-03-29T11:26:52.205 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.206+0000 7febd3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:52.205 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.206+0000 7febd3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:52.498 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.512+0000 7febf21ca640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:52.499 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.512+0000 7febf21ca640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:52.499 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.512+0000 7febf21ca640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:26:52.499 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.512+0000 7febf21ca640 1 -- 172.21.15.135:0/2172897442 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febcc005330 con 0x7febec0b1f60 2024-03-29T11:26:52.499 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.513+0000 7febd3fff640 1 -- 172.21.15.135:0/2172897442 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7febdc051040 con 0x7febec0b1f60 2024-03-29T11:26:52.499 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.513+0000 7febd3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:52.499 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.513+0000 7febd3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:52.500 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:26:52.500 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:52.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.516+0000 7febf21ca640 10 librados: watch_flush enter 2024-03-29T11:26:52.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.516+0000 7febf21ca640 10 librados: watch_flush exit 2024-03-29T11:26:52.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.516+0000 7febf21ca640 20 client.4318.objecter shutdown clearing up homeless session... 2024-03-29T11:26:52.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.516+0000 7febf21ca640 10 client.4318.objecter successfully canceled tick 2024-03-29T11:26:52.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.516+0000 7febf21ca640 1 -- 172.21.15.135:0/2172897442 >> v1:172.21.15.135:6816/3469875285 conn(0x7febd806dd60 legacy=0x7febd8070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:52.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.516+0000 7febf21ca640 10 monclient: shutdown 2024-03-29T11:26:52.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.516+0000 7febf21ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:52.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.516+0000 7febf21ca640 1 -- 172.21.15.135:0/2172897442 >> v1:172.21.15.194:6789/0 conn(0x7febec0b1f60 legacy=0x7febec13c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:52.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.516+0000 7febf21ca640 1 -- 172.21.15.135:0/2172897442 shutdown_connections 2024-03-29T11:26:52.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.516+0000 7febf21ca640 1 -- 172.21.15.135:0/2172897442 >> 172.21.15.135:0/2172897442 conn(0x7febec066a00 msgr2=0x7febec066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:52.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.516+0000 7febf21ca640 1 -- 172.21.15.135:0/2172897442 shutdown_connections 2024-03-29T11:26:52.504 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.516+0000 7febf21ca640 1 -- 172.21.15.135:0/2172897442 wait complete. 2024-03-29T11:26:52.504 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:52.516+0000 7febf21ca640 1 librados: shutdown 2024-03-29T11:26:52.518 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:59.326 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:59.473 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.484+0000 7ff00bfff640 10 monclient: get_monmap_and_config 2024-03-29T11:26:59.473 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.484+0000 7ff00bfff640 10 monclient: build_initial_monmap 2024-03-29T11:26:59.473 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.484+0000 7ff00bfff640 10 monclient: monmap: 2024-03-29T11:26:59.473 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:26:59.473 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:59.473 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:26:59.485468+0000 2024-03-29T11:26:59.473 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:26:59.485468+0000 2024-03-29T11:26:59.474 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:59.474 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:59.474 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:26:59.474 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:26:59.474 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:26:59.474 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:59.474 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.485+0000 7ff00bfff640 1 Processor -- start 2024-03-29T11:26:59.474 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.485+0000 7ff00bfff640 1 -- start start 2024-03-29T11:26:59.474 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.485+0000 7ff00bfff640 10 monclient: init 2024-03-29T11:26:59.481 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.486+0000 7ff00bfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:59.481 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.486+0000 7ff00bfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:59.481 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.486+0000 7ff00bfff640 10 monclient(hunting): picked mon.noname-b con 0x7ff00402b090 addr v1:172.21.15.135:6790/0 2024-03-29T11:26:59.481 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.486+0000 7ff00bfff640 10 monclient(hunting): picked mon.noname-a con 0x7ff004006940 addr v1:172.21.15.135:6789/0 2024-03-29T11:26:59.481 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.486+0000 7ff00bfff640 10 monclient(hunting): picked mon.noname-c con 0x7ff0040e2530 addr v1:172.21.15.194:6789/0 2024-03-29T11:26:59.481 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.486+0000 7ff00bfff640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff004006d80 con 0x7ff004006940 2024-03-29T11:26:59.481 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.486+0000 7ff00bfff640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff00402a430 con 0x7ff0040e2530 2024-03-29T11:26:59.481 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.486+0000 7ff00bfff640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff00402a5c0 con 0x7ff00402b090 2024-03-29T11:26:59.482 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.486+0000 7ff00bfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:59.482 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.486+0000 7ff00bfff640 10 monclient(hunting): authenticate will time out at 926.854908s 2024-03-29T11:26:59.482 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.487+0000 7ff00b7fe640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7ff0040e2530 0x7ff0040e29c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:60410/0 (socket says 172.21.15.135:60410) 2024-03-29T11:26:59.482 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.487+0000 7ff00b7fe640 1 -- 172.21.15.135:0/4126701401 learned_addr learned my addr 172.21.15.135:0/4126701401 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:26:59.482 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.488+0000 7ff009ffb640 1 -- 172.21.15.135:0/4126701401 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802788247 0 0) 0x7ff00402a430 con 0x7ff0040e2530 2024-03-29T11:26:59.482 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.488+0000 7ff009ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:59.482 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.488+0000 7ff009ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:59.482 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.488+0000 7ff009ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:59.482 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.488+0000 7ff009ffb640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:26:59.482 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.488+0000 7ff009ffb640 1 -- 172.21.15.135:0/4126701401 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feff400bd30 con 0x7ff0040e2530 2024-03-29T11:26:59.482 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.489+0000 7ff009ffb640 1 -- 172.21.15.135:0/4126701401 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1776293963 0 0) 0x7feff400bd30 con 0x7ff0040e2530 2024-03-29T11:26:59.482 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.489+0000 7ff009ffb640 1 -- 172.21.15.135:0/4126701401 >> v1:172.21.15.135:6790/0 conn(0x7ff00402b090 legacy=0x7ff0040e8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:59.482 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.489+0000 7ff009ffb640 1 -- 172.21.15.135:0/4126701401 >> v1:172.21.15.135:6789/0 conn(0x7ff004006940 legacy=0x7ff004029c30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:59.483 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.493+0000 7ff009ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:59.483 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.493+0000 7ff009ffb640 1 monclient: found mon.noname-c 2024-03-29T11:26:59.483 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.493+0000 7ff009ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:59.483 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.493+0000 7ff009ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:26:59.483 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.493+0000 7ff009ffb640 1 -- 172.21.15.135:0/4126701401 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0040edd90 con 0x7ff0040e2530 2024-03-29T11:26:59.483 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.493+0000 7ff009ffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:59.483 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.493+0000 7ff009ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:59.483 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.493+0000 7ff009ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:59.483 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.493+0000 7ff00bfff640 5 monclient: authenticate success, global_id 4321 2024-03-29T11:26:59.483 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.493+0000 7ff00bfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:59.483 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.493+0000 7ff009ffb640 1 -- 172.21.15.135:0/4126701401 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7ff00c04f730 con 0x7ff0040e2530 2024-03-29T11:26:59.483 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.493+0000 7ff009ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:59.483 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.493+0000 7ff009ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:59.483 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.493+0000 7ff009ffb640 10 monclient: dump: 2024-03-29T11:26:59.483 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:59.484 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:59.484 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:59.484 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:59.484 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:59.484 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:59.484 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:59.484 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:59.484 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:59.484 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:59.484 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.493+0000 7ff00bfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:59.485 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.494+0000 7ff009ffb640 1 -- 172.21.15.135:0/4126701401 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff00c0008d0 con 0x7ff0040e2530 2024-03-29T11:26:59.486 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.494+0000 7ff009ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:59.486 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.494+0000 7ff00bfff640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:59.486 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.494+0000 7ff00bfff640 4 set_mon_vals no callback set 2024-03-29T11:26:59.486 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.495+0000 7ff00bfff640 10 monclient: shutdown 2024-03-29T11:26:59.486 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.495+0000 7ff00bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:59.486 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.495+0000 7ff00bfff640 1 -- 172.21.15.135:0/4126701401 >> v1:172.21.15.194:6789/0 conn(0x7ff0040e2530 legacy=0x7ff0040e29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:59.486 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.497+0000 7ff00bfff640 1 -- 172.21.15.135:0/4126701401 shutdown_connections 2024-03-29T11:26:59.486 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.497+0000 7ff00bfff640 1 -- 172.21.15.135:0/4126701401 >> 172.21.15.135:0/4126701401 conn(0x7ff0040058f0 msgr2=0x7ff004005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:59.487 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.499+0000 7ff00bfff640 1 -- 172.21.15.135:0/4126701401 shutdown_connections 2024-03-29T11:26:59.487 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.499+0000 7ff00bfff640 1 -- 172.21.15.135:0/4126701401 wait complete. 2024-03-29T11:26:59.489 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.500+0000 7ff00bfff640 10 monclient: build_initial_monmap 2024-03-29T11:26:59.489 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.500+0000 7ff00bfff640 10 monclient: monmap: 2024-03-29T11:26:59.489 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:26:59.490 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:59.490 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:26:59.490 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:26:59.490 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:59.490 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:59.490 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:26:59.490 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:26:59.490 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:26:59.490 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:59.490 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.501+0000 7ff00bfff640 1 librados: starting msgr at 2024-03-29T11:26:59.490 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.501+0000 7ff00bfff640 1 librados: starting objecter 2024-03-29T11:26:59.490 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.501+0000 7ff00bfff640 1 Processor -- start 2024-03-29T11:26:59.491 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.501+0000 7ff00bfff640 1 -- start start 2024-03-29T11:26:59.491 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.501+0000 7ff00bfff640 1 librados: setting wanted keys 2024-03-29T11:26:59.491 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.501+0000 7ff00bfff640 1 librados: calling monclient init 2024-03-29T11:26:59.491 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.501+0000 7ff00bfff640 10 monclient: init 2024-03-29T11:26:59.491 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.502+0000 7ff00bfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:59.492 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.502+0000 7ff00bfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:59.492 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.502+0000 7ff00bfff640 10 monclient(hunting): picked mon.noname-a con 0x7ff004006940 addr v1:172.21.15.135:6789/0 2024-03-29T11:26:59.492 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.502+0000 7ff00bfff640 10 monclient(hunting): picked mon.noname-b con 0x7ff00402b090 addr v1:172.21.15.194:6789/0 2024-03-29T11:26:59.492 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.502+0000 7ff00bfff640 10 monclient(hunting): picked mon.noname-c con 0x7ff0040e2530 addr v1:172.21.15.135:6790/0 2024-03-29T11:26:59.492 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.502+0000 7ff00bfff640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0040dcc00 con 0x7ff004006940 2024-03-29T11:26:59.492 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.502+0000 7ff00bfff640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff004035170 con 0x7ff00402b090 2024-03-29T11:26:59.492 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.502+0000 7ff00bfff640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0040d67f0 con 0x7ff0040e2530 2024-03-29T11:26:59.492 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.502+0000 7ff00bfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:59.492 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.502+0000 7ff00bfff640 10 monclient(hunting): authenticate will time out at 926.870408s 2024-03-29T11:26:59.493 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.505+0000 7ff00affd640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7ff004006940 0x7ff004034a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:53463/0 (socket says 172.21.15.135:53463) 2024-03-29T11:26:59.493 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.505+0000 7ff00affd640 1 -- 172.21.15.135:0/2944672926 learned_addr learned my addr 172.21.15.135:0/2944672926 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:26:59.497 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.505+0000 7feff37fe640 10 client.?.objecter ms_handle_connect 0x7ff004006940 2024-03-29T11:26:59.497 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.505+0000 7feff37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:59.497 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.505+0000 7feff37fe640 1 -- 172.21.15.135:0/2944672926 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1388339771 0 0) 0x7ff0040dcc00 con 0x7ff004006940 2024-03-29T11:26:59.497 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.505+0000 7feff37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:59.497 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.505+0000 7feff37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:59.497 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.505+0000 7feff37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:59.497 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.505+0000 7feff37fe640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:26:59.497 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.505+0000 7feff37fe640 1 -- 172.21.15.135:0/2944672926 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feff8004eb0 con 0x7ff004006940 2024-03-29T11:26:59.497 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.506+0000 7feff37fe640 1 -- 172.21.15.135:0/2944672926 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 932447158 0 0) 0x7feff8004eb0 con 0x7ff004006940 2024-03-29T11:26:59.497 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.506+0000 7feff37fe640 1 -- 172.21.15.135:0/2944672926 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0040dcc00 con 0x7ff004006940 2024-03-29T11:26:59.497 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.506+0000 7feff37fe640 1 -- 172.21.15.135:0/2944672926 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7ff000001960 con 0x7ff004006940 2024-03-29T11:26:59.498 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.506+0000 7feff37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:59.498 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.506+0000 7feff37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:59.498 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.506+0000 7feff37fe640 10 monclient(hunting): dump: 2024-03-29T11:26:59.498 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:59.498 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:59.498 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:59.498 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:59.498 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:59.498 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:59.498 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:59.498 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:59.498 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:59.498 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:59.498 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.506+0000 7feff37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:59.499 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.506+0000 7feff37fe640 1 -- 172.21.15.135:0/2944672926 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3920664340 0 0) 0x7ff0040dcc00 con 0x7ff004006940 2024-03-29T11:26:59.499 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.506+0000 7feff37fe640 1 -- 172.21.15.135:0/2944672926 >> v1:172.21.15.135:6790/0 conn(0x7ff0040e2530 legacy=0x7ff0040d60e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:59.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.507+0000 7feff37fe640 1 -- 172.21.15.135:0/2944672926 >> v1:172.21.15.194:6789/0 conn(0x7ff00402b090 legacy=0x7ff0040d59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:59.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.507+0000 7feff37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:59.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.507+0000 7feff37fe640 1 monclient: found mon.a 2024-03-29T11:26:59.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.507+0000 7feff37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:59.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.507+0000 7feff37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:26:59.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.507+0000 7feff37fe640 1 -- 172.21.15.135:0/2944672926 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0040dd090 con 0x7ff004006940 2024-03-29T11:26:59.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.507+0000 7feff37fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:59.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.507+0000 7feff37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:59.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.507+0000 7feff37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:59.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.507+0000 7feff37fe640 10 client.?.objecter ms_handle_connect 0x7ff00402b090 2024-03-29T11:26:59.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.507+0000 7feff37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:59.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.507+0000 7ff00bfff640 5 monclient: authenticate success, global_id 4329 2024-03-29T11:26:59.504 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.507+0000 7ff00bfff640 10 monclient: _renew_subs 2024-03-29T11:26:59.504 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.507+0000 7ff00bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:26:59.504 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.507+0000 7ff00bfff640 1 -- 172.21.15.135:0/2944672926 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0040deb20 con 0x7ff004006940 2024-03-29T11:26:59.504 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.507+0000 7ff00bfff640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:59.504 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.507+0000 7ff00bfff640 10 monclient: _renew_subs 2024-03-29T11:26:59.504 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.507+0000 7ff00bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:26:59.504 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.507+0000 7ff00bfff640 1 -- 172.21.15.135:0/2944672926 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff004038a40 con 0x7ff004006940 2024-03-29T11:26:59.504 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.507+0000 7ff00bfff640 1 librados: init done 2024-03-29T11:26:59.505 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.509+0000 7ff00bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:59.505 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.509+0000 7ff00bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:59.505 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.509+0000 7ff00bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:26:59.505 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.509+0000 7feff37fe640 1 -- 172.21.15.135:0/2944672926 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff000001c80 con 0x7ff004006940 2024-03-29T11:26:59.505 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.509+0000 7ff00bfff640 1 -- 172.21.15.135:0/2944672926 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff004004cd0 con 0x7ff004006940 2024-03-29T11:26:59.505 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.509+0000 7feff37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:59.505 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.509+0000 7feff37fe640 1 -- 172.21.15.135:0/2944672926 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7ff000003a40 con 0x7ff004006940 2024-03-29T11:26:59.505 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.509+0000 7feff37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:59.505 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.509+0000 7feff37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:59.506 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.509+0000 7feff37fe640 10 monclient: dump: 2024-03-29T11:26:59.506 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:26:59.506 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:26:59.506 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:59.506 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:26:59.506 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:59.506 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:26:59.506 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:26:59.506 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:26:59.506 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:26:59.506 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:26:59.506 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.509+0000 7ff0097fa640 4 set_mon_vals no callback set 2024-03-29T11:26:59.506 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.511+0000 7feff37fe640 1 -- 172.21.15.135:0/2944672926 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7ff000003ba0 con 0x7ff004006940 2024-03-29T11:26:59.507 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.512+0000 7feff37fe640 1 -- 172.21.15.135:0/2944672926 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7ff000084ae0 con 0x7ff004006940 2024-03-29T11:26:59.507 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.512+0000 7feff37fe640 10 client.4329.objecter ms_dispatch 0x7ff00c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:59.507 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.512+0000 7feff37fe640 3 client.4329.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:59.507 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.512+0000 7feff37fe640 3 client.4329.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:59.507 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.512+0000 7feff37fe640 20 client.4329.objecter dump_active .. 0 homeless 2024-03-29T11:26:59.507 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.512+0000 7ff00a7fc640 10 monclient: get_auth_request con 0x7feff806dc00 auth_method 0 2024-03-29T11:26:59.507 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.512+0000 7feff37fe640 10 client.4329.objecter ms_handle_connect 0x7feff806dc00 2024-03-29T11:26:59.508 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.519+0000 7feff37fe640 1 -- 172.21.15.135:0/2944672926 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff0000503c0 con 0x7ff004006940 2024-03-29T11:26:59.508 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.519+0000 7feff37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:59.508 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.519+0000 7feff37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:59.836 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.849+0000 7ff00bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:59.836 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.849+0000 7ff00bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:59.836 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.849+0000 7ff00bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:26:59.836 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.849+0000 7ff00bfff640 1 -- 172.21.15.135:0/2944672926 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0040ded60 con 0x7ff004006940 2024-03-29T11:26:59.839 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.853+0000 7feff37fe640 1 -- 172.21.15.135:0/2944672926 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7ff0000522f0 con 0x7ff004006940 2024-03-29T11:26:59.839 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.853+0000 7feff37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:59.840 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.853+0000 7feff37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:59.840 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:26:59.840 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:59.844 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.857+0000 7ff00bfff640 10 librados: watch_flush enter 2024-03-29T11:26:59.844 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.857+0000 7ff00bfff640 10 librados: watch_flush exit 2024-03-29T11:26:59.844 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.857+0000 7ff00bfff640 20 client.4329.objecter shutdown clearing up homeless session... 2024-03-29T11:26:59.844 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.857+0000 7ff00bfff640 10 client.4329.objecter successfully canceled tick 2024-03-29T11:26:59.845 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.858+0000 7ff00bfff640 1 -- 172.21.15.135:0/2944672926 >> v1:172.21.15.135:6816/3469875285 conn(0x7feff806dc00 legacy=0x7feff80700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:59.845 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.859+0000 7ff00bfff640 10 monclient: shutdown 2024-03-29T11:26:59.845 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.859+0000 7ff00bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:59.845 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.859+0000 7ff00bfff640 1 -- 172.21.15.135:0/2944672926 >> v1:172.21.15.135:6789/0 conn(0x7ff004006940 legacy=0x7ff004034a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:59.852 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.860+0000 7ff00bfff640 1 -- 172.21.15.135:0/2944672926 shutdown_connections 2024-03-29T11:26:59.853 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.860+0000 7ff00bfff640 1 -- 172.21.15.135:0/2944672926 >> 172.21.15.135:0/2944672926 conn(0x7ff0040058f0 msgr2=0x7ff0040dc320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:59.853 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.860+0000 7ff00bfff640 1 -- 172.21.15.135:0/2944672926 shutdown_connections 2024-03-29T11:26:59.853 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.862+0000 7ff00bfff640 1 -- 172.21.15.135:0/2944672926 wait complete. 2024-03-29T11:26:59.853 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:26:59.862+0000 7ff00bfff640 1 librados: shutdown 2024-03-29T11:26:59.870 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:06.681 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:06.835 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.847+0000 7f5d7766a640 10 monclient: get_monmap_and_config 2024-03-29T11:27:06.835 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.847+0000 7f5d7766a640 10 monclient: build_initial_monmap 2024-03-29T11:27:06.835 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.847+0000 7f5d7766a640 10 monclient: monmap: 2024-03-29T11:27:06.835 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:27:06.835 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:06.835 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:27:06.847695+0000 2024-03-29T11:27:06.835 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:27:06.847695+0000 2024-03-29T11:27:06.835 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:06.836 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:06.836 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:27:06.836 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:27:06.836 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:27:06.836 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:06.836 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.848+0000 7f5d7766a640 1 Processor -- start 2024-03-29T11:27:06.838 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.849+0000 7f5d7766a640 1 -- start start 2024-03-29T11:27:06.838 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.849+0000 7f5d7766a640 10 monclient: init 2024-03-29T11:27:06.838 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.849+0000 7f5d7766a640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:06.838 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.849+0000 7f5d7766a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:27:06.838 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.849+0000 7f5d7766a640 10 monclient(hunting): picked mon.noname-b con 0x7f5d700a9ac0 addr v1:172.21.15.135:6790/0 2024-03-29T11:27:06.838 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.849+0000 7f5d7766a640 10 monclient(hunting): picked mon.noname-c con 0x7f5d7013ee90 addr v1:172.21.15.194:6789/0 2024-03-29T11:27:06.838 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.850+0000 7f5d7766a640 10 monclient(hunting): picked mon.noname-a con 0x7f5d700af900 addr v1:172.21.15.135:6789/0 2024-03-29T11:27:06.839 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.850+0000 7f5d7766a640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d700afd10 con 0x7f5d700af900 2024-03-29T11:27:06.839 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.850+0000 7f5d7766a640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d700af6f0 con 0x7f5d7013ee90 2024-03-29T11:27:06.839 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.850+0000 7f5d7766a640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d700aac80 con 0x7f5d700a9ac0 2024-03-29T11:27:06.839 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.850+0000 7f5d7766a640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:06.839 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.850+0000 7f5d7766a640 10 monclient(hunting): authenticate will time out at 934.218089s 2024-03-29T11:27:06.844 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.857+0000 7f5d75c07640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f5d700af900 0x7f5d70149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:58345/0 (socket says 172.21.15.135:58345) 2024-03-29T11:27:06.844 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.857+0000 7f5d75c07640 1 -- 172.21.15.135:0/812152337 learned_addr learned my addr 172.21.15.135:0/812152337 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:27:06.847 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.859+0000 7f5d67fff640 1 -- 172.21.15.135:0/812152337 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 143608841 0 0) 0x7f5d700aac80 con 0x7f5d700a9ac0 2024-03-29T11:27:06.847 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.859+0000 7f5d67fff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:06.847 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.859+0000 7f5d67fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:06.847 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.859+0000 7f5d67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:06.847 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.859+0000 7f5d67fff640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:27:06.848 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.859+0000 7f5d67fff640 1 -- 172.21.15.135:0/812152337 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d5c003520 con 0x7f5d700a9ac0 2024-03-29T11:27:06.848 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.859+0000 7f5d67fff640 1 -- 172.21.15.135:0/812152337 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821277049 0 0) 0x7f5d700afd10 con 0x7f5d700af900 2024-03-29T11:27:06.848 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.859+0000 7f5d67fff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:06.848 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.859+0000 7f5d67fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:06.848 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.859+0000 7f5d67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:06.848 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.859+0000 7f5d67fff640 10 monclient(hunting): my global_id is 4332 2024-03-29T11:27:06.848 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.859+0000 7f5d67fff640 1 -- 172.21.15.135:0/812152337 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d700aac80 con 0x7f5d700af900 2024-03-29T11:27:06.848 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.860+0000 7f5d67fff640 1 -- 172.21.15.135:0/812152337 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1629894979 0 0) 0x7f5d5c003520 con 0x7f5d700a9ac0 2024-03-29T11:27:06.849 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.860+0000 7f5d67fff640 1 -- 172.21.15.135:0/812152337 >> v1:172.21.15.194:6789/0 conn(0x7f5d7013ee90 legacy=0x7f5d7013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:06.849 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.860+0000 7f5d67fff640 1 -- 172.21.15.135:0/812152337 >> v1:172.21.15.135:6789/0 conn(0x7f5d700af900 legacy=0x7f5d70149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:06.849 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.860+0000 7f5d67fff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:06.849 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.860+0000 7f5d67fff640 1 monclient: found mon.noname-b 2024-03-29T11:27:06.849 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.860+0000 7f5d67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:06.849 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.860+0000 7f5d67fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:27:06.849 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.860+0000 7f5d67fff640 1 -- 172.21.15.135:0/812152337 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d700aaf60 con 0x7f5d700a9ac0 2024-03-29T11:27:06.849 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.860+0000 7f5d67fff640 10 monclient: _finish_auth 0 2024-03-29T11:27:06.849 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.860+0000 7f5d67fff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:06.850 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.860+0000 7f5d67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:06.850 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.860+0000 7f5d67fff640 1 -- 172.21.15.135:0/812152337 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f5d6c001690 con 0x7f5d700a9ac0 2024-03-29T11:27:06.850 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.860+0000 7f5d67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:06.850 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.860+0000 7f5d67fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:06.850 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.860+0000 7f5d67fff640 10 monclient: dump: 2024-03-29T11:27:06.850 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:06.850 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:06.850 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:06.850 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:06.851 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:06.851 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:06.851 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:06.851 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:06.851 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:06.851 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:06.851 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.860+0000 7f5d7766a640 5 monclient: authenticate success, global_id 4319 2024-03-29T11:27:06.851 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.860+0000 7f5d7766a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:06.851 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.860+0000 7f5d67fff640 1 -- 172.21.15.135:0/812152337 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d6c0017f0 con 0x7f5d700a9ac0 2024-03-29T11:27:06.851 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.860+0000 7f5d67fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:06.852 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.860+0000 7f5d67fff640 1 -- 172.21.15.135:0/812152337 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f5d6c001c00 con 0x7f5d700a9ac0 2024-03-29T11:27:06.852 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.860+0000 7f5d67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:06.852 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.860+0000 7f5d67fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:06.852 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.860+0000 7f5d67fff640 10 monclient: dump: 2024-03-29T11:27:06.852 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:06.852 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:06.852 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:06.852 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:06.852 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:06.852 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:06.853 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:06.853 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:06.853 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:06.853 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:06.855 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.861+0000 7f5d7766a640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:06.855 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.861+0000 7f5d7766a640 4 set_mon_vals no callback set 2024-03-29T11:27:06.855 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.862+0000 7f5d7766a640 10 monclient: shutdown 2024-03-29T11:27:06.855 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.862+0000 7f5d7766a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:06.856 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.862+0000 7f5d7766a640 1 -- 172.21.15.135:0/812152337 >> v1:172.21.15.135:6790/0 conn(0x7f5d700a9ac0 legacy=0x7f5d700adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:06.856 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.862+0000 7f5d7766a640 1 -- 172.21.15.135:0/812152337 shutdown_connections 2024-03-29T11:27:06.856 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.862+0000 7f5d7766a640 1 -- 172.21.15.135:0/812152337 >> 172.21.15.135:0/812152337 conn(0x7f5d700b35e0 msgr2=0x7f5d700b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:06.856 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.863+0000 7f5d7766a640 1 -- 172.21.15.135:0/812152337 shutdown_connections 2024-03-29T11:27:06.856 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.863+0000 7f5d7766a640 1 -- 172.21.15.135:0/812152337 wait complete. 2024-03-29T11:27:06.856 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.863+0000 7f5d7766a640 10 monclient: build_initial_monmap 2024-03-29T11:27:06.856 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.863+0000 7f5d7766a640 10 monclient: monmap: 2024-03-29T11:27:06.856 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:27:06.856 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:06.856 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:27:06.856 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:27:06.856 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:06.856 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:06.856 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:27:06.856 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:27:06.857 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:27:06.857 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:06.857 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.864+0000 7f5d7766a640 1 librados: starting msgr at 2024-03-29T11:27:06.857 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.864+0000 7f5d7766a640 1 librados: starting objecter 2024-03-29T11:27:06.857 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.864+0000 7f5d7766a640 1 Processor -- start 2024-03-29T11:27:06.857 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.864+0000 7f5d7766a640 1 -- start start 2024-03-29T11:27:06.857 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.864+0000 7f5d7766a640 1 librados: setting wanted keys 2024-03-29T11:27:06.857 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.864+0000 7f5d7766a640 1 librados: calling monclient init 2024-03-29T11:27:06.857 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.864+0000 7f5d7766a640 10 monclient: init 2024-03-29T11:27:06.857 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.865+0000 7f5d7766a640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:06.857 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.865+0000 7f5d7766a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:06.857 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.865+0000 7f5d7766a640 10 monclient(hunting): picked mon.noname-b con 0x7f5d700a9ac0 addr v1:172.21.15.194:6789/0 2024-03-29T11:27:06.857 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.865+0000 7f5d7766a640 10 monclient(hunting): picked mon.noname-a con 0x7f5d700af900 addr v1:172.21.15.135:6789/0 2024-03-29T11:27:06.857 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.865+0000 7f5d7766a640 10 monclient(hunting): picked mon.noname-c con 0x7f5d7013ee90 addr v1:172.21.15.135:6790/0 2024-03-29T11:27:06.857 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.865+0000 7f5d7766a640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d701ded20 con 0x7f5d700af900 2024-03-29T11:27:06.858 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.865+0000 7f5d7766a640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7011ca20 con 0x7f5d700a9ac0 2024-03-29T11:27:06.858 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.865+0000 7f5d7766a640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7011cd00 con 0x7f5d7013ee90 2024-03-29T11:27:06.858 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.865+0000 7f5d7766a640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:06.858 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.865+0000 7f5d7766a640 10 monclient(hunting): authenticate will time out at 934.233474s 2024-03-29T11:27:06.859 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.869+0000 7f5d75c07640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f5d7013ee90 0x7f5d7013a350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:55705/0 (socket says 172.21.15.135:55705) 2024-03-29T11:27:06.859 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.869+0000 7f5d75c07640 1 -- 172.21.15.135:0/737098803 learned_addr learned my addr 172.21.15.135:0/737098803 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:27:06.859 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.870+0000 7f5d65ffb640 10 client.?.objecter ms_handle_connect 0x7f5d7013ee90 2024-03-29T11:27:06.859 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.871+0000 7f5d65ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:06.859 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.871+0000 7f5d65ffb640 10 client.?.objecter ms_handle_connect 0x7f5d700af900 2024-03-29T11:27:06.859 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.871+0000 7f5d65ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:06.859 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.871+0000 7f5d65ffb640 10 client.?.objecter ms_handle_connect 0x7f5d700a9ac0 2024-03-29T11:27:06.859 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.871+0000 7f5d65ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:06.860 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.872+0000 7f5d65ffb640 1 -- 172.21.15.135:0/737098803 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1703187334 0 0) 0x7f5d7011ca20 con 0x7f5d700a9ac0 2024-03-29T11:27:06.860 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.872+0000 7f5d65ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:06.860 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.872+0000 7f5d65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:06.860 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.872+0000 7f5d65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:06.860 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.872+0000 7f5d65ffb640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:27:06.860 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.872+0000 7f5d65ffb640 1 -- 172.21.15.135:0/737098803 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d54004eb0 con 0x7f5d700a9ac0 2024-03-29T11:27:06.860 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.872+0000 7f5d65ffb640 1 -- 172.21.15.135:0/737098803 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2005724650 0 0) 0x7f5d7011cd00 con 0x7f5d7013ee90 2024-03-29T11:27:06.860 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.873+0000 7f5d65ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:06.861 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.873+0000 7f5d65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:06.861 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.873+0000 7f5d65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:06.861 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.873+0000 7f5d65ffb640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:27:06.861 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.873+0000 7f5d65ffb640 1 -- 172.21.15.135:0/737098803 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d7011ca20 con 0x7f5d7013ee90 2024-03-29T11:27:06.861 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.873+0000 7f5d65ffb640 1 -- 172.21.15.135:0/737098803 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222094857 0 0) 0x7f5d701ded20 con 0x7f5d700af900 2024-03-29T11:27:06.863 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.873+0000 7f5d65ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:06.863 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.874+0000 7f5d65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:06.863 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.874+0000 7f5d65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:06.863 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.874+0000 7f5d65ffb640 10 monclient(hunting): my global_id is 4335 2024-03-29T11:27:06.864 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.874+0000 7f5d65ffb640 1 -- 172.21.15.135:0/737098803 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d7011cd00 con 0x7f5d700af900 2024-03-29T11:27:06.864 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.874+0000 7f5d65ffb640 1 -- 172.21.15.135:0/737098803 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1286835866 0 0) 0x7f5d54004eb0 con 0x7f5d700a9ac0 2024-03-29T11:27:06.864 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.874+0000 7f5d65ffb640 1 -- 172.21.15.135:0/737098803 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d701ded20 con 0x7f5d700a9ac0 2024-03-29T11:27:06.864 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.874+0000 7f5d65ffb640 1 -- 172.21.15.135:0/737098803 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1436304857 0 0) 0x7f5d7011ca20 con 0x7f5d7013ee90 2024-03-29T11:27:06.864 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.874+0000 7f5d65ffb640 1 -- 172.21.15.135:0/737098803 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d54004eb0 con 0x7f5d7013ee90 2024-03-29T11:27:06.864 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.874+0000 7f5d65ffb640 1 -- 172.21.15.135:0/737098803 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f5d6c001bc0 con 0x7f5d700a9ac0 2024-03-29T11:27:06.864 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.874+0000 7f5d65ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:06.864 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.874+0000 7f5d65ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:06.864 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.874+0000 7f5d65ffb640 10 monclient(hunting): dump: 2024-03-29T11:27:06.864 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:06.864 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:06.865 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:06.865 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:06.865 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:06.865 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:06.865 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:06.865 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:06.865 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:06.865 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:06.865 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.875+0000 7f5d65ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:06.865 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.875+0000 7f5d65ffb640 1 -- 172.21.15.135:0/737098803 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f5d68001c30 con 0x7f5d7013ee90 2024-03-29T11:27:06.865 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.875+0000 7f5d65ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:06.865 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.875+0000 7f5d65ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:06.865 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.875+0000 7f5d65ffb640 10 monclient(hunting): dump: 2024-03-29T11:27:06.866 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:06.866 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:06.866 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:06.866 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:06.866 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:06.866 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:06.866 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:06.866 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:06.866 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:06.866 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:06.866 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.876+0000 7f5d65ffb640 1 -- 172.21.15.135:0/737098803 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3670384857 0 0) 0x7f5d54004eb0 con 0x7f5d7013ee90 2024-03-29T11:27:06.867 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.876+0000 7f5d65ffb640 1 -- 172.21.15.135:0/737098803 >> v1:172.21.15.194:6789/0 conn(0x7f5d700a9ac0 legacy=0x7f5d7011b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:06.867 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.876+0000 7f5d65ffb640 1 -- 172.21.15.135:0/737098803 >> v1:172.21.15.135:6789/0 conn(0x7f5d700af900 legacy=0x7f5d701df200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:06.867 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.876+0000 7f5d65ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:06.867 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.876+0000 7f5d65ffb640 1 monclient: found mon.c 2024-03-29T11:27:06.867 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.876+0000 7f5d65ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:06.867 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.876+0000 7f5d65ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:27:06.867 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.876+0000 7f5d65ffb640 1 -- 172.21.15.135:0/737098803 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7006f7f0 con 0x7f5d7013ee90 2024-03-29T11:27:06.867 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.876+0000 7f5d65ffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:06.867 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.876+0000 7f5d65ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:06.867 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.876+0000 7f5d65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:06.867 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.876+0000 7f5d7766a640 5 monclient: authenticate success, global_id 4322 2024-03-29T11:27:06.867 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.876+0000 7f5d7766a640 10 monclient: _renew_subs 2024-03-29T11:27:06.867 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.876+0000 7f5d7766a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:27:06.867 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.876+0000 7f5d7766a640 1 -- 172.21.15.135:0/737098803 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d701df910 con 0x7f5d7013ee90 2024-03-29T11:27:06.867 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.876+0000 7f5d7766a640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:06.868 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.876+0000 7f5d7766a640 10 monclient: _renew_subs 2024-03-29T11:27:06.868 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.876+0000 7f5d7766a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:27:06.868 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.876+0000 7f5d7766a640 1 -- 172.21.15.135:0/737098803 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7013ac40 con 0x7f5d7013ee90 2024-03-29T11:27:06.868 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.876+0000 7f5d7766a640 1 librados: init done 2024-03-29T11:27:06.868 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.876+0000 7f5d65ffb640 1 -- 172.21.15.135:0/737098803 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d68004b60 con 0x7f5d7013ee90 2024-03-29T11:27:06.868 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.876+0000 7f5d65ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:06.868 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.876+0000 7f5d65ffb640 1 -- 172.21.15.135:0/737098803 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f5d68004f30 con 0x7f5d7013ee90 2024-03-29T11:27:06.868 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.876+0000 7f5d65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:06.868 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.876+0000 7f5d66ffd640 4 set_mon_vals no callback set 2024-03-29T11:27:06.868 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.877+0000 7f5d65ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:06.868 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.877+0000 7f5d65ffb640 10 monclient: dump: 2024-03-29T11:27:06.868 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:06.868 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:06.869 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:06.869 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:06.869 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:06.869 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:06.869 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:06.869 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:06.869 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:06.869 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:06.870 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.879+0000 7f5d7766a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:06.870 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.879+0000 7f5d7766a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:06.870 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.879+0000 7f5d7766a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:27:06.870 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.879+0000 7f5d7766a640 1 -- 172.21.15.135:0/737098803 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d50005080 con 0x7f5d7013ee90 2024-03-29T11:27:06.870 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.884+0000 7f5d65ffb640 1 -- 172.21.15.135:0/737098803 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f5d68005090 con 0x7f5d7013ee90 2024-03-29T11:27:06.871 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.884+0000 7f5d65ffb640 1 -- 172.21.15.135:0/737098803 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f5d68084cc0 con 0x7f5d7013ee90 2024-03-29T11:27:06.872 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.885+0000 7f5d65ffb640 10 client.4322.objecter ms_dispatch 0x7f5d70000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:06.872 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.885+0000 7f5d65ffb640 3 client.4322.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:06.872 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.885+0000 7f5d65ffb640 3 client.4322.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:06.872 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.885+0000 7f5d65ffb640 20 client.4322.objecter dump_active .. 0 homeless 2024-03-29T11:27:06.872 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.885+0000 7f5d75406640 10 monclient: get_auth_request con 0x7f5d5406e030 auth_method 0 2024-03-29T11:27:06.872 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.886+0000 7f5d65ffb640 10 client.4322.objecter ms_handle_connect 0x7f5d5406e030 2024-03-29T11:27:06.878 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.890+0000 7f5d65ffb640 1 -- 172.21.15.135:0/737098803 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5d680505a0 con 0x7f5d7013ee90 2024-03-29T11:27:06.878 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.890+0000 7f5d65ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:06.878 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:06.890+0000 7f5d65ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:07.208 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:07.221+0000 7f5d7766a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:07.208 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:07.221+0000 7f5d7766a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:07.208 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:07.221+0000 7f5d7766a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:27:07.209 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:07.221+0000 7f5d7766a640 1 -- 172.21.15.135:0/737098803 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d50005330 con 0x7f5d7013ee90 2024-03-29T11:27:07.209 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:07.222+0000 7f5d65ffb640 1 -- 172.21.15.135:0/737098803 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f5d680524d0 con 0x7f5d7013ee90 2024-03-29T11:27:07.209 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:07.222+0000 7f5d65ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:07.210 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:27:07.210 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:07.210 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:07.222+0000 7f5d65ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:07.213 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:07.226+0000 7f5d7766a640 10 librados: watch_flush enter 2024-03-29T11:27:07.214 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:07.226+0000 7f5d7766a640 10 librados: watch_flush exit 2024-03-29T11:27:07.214 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:07.226+0000 7f5d7766a640 20 client.4322.objecter shutdown clearing up homeless session... 2024-03-29T11:27:07.214 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:07.226+0000 7f5d7766a640 10 client.4322.objecter successfully canceled tick 2024-03-29T11:27:07.214 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:07.227+0000 7f5d7766a640 1 -- 172.21.15.135:0/737098803 >> v1:172.21.15.135:6816/3469875285 conn(0x7f5d5406e030 legacy=0x7f5d540704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:07.214 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:07.227+0000 7f5d7766a640 10 monclient: shutdown 2024-03-29T11:27:07.214 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:07.227+0000 7f5d7766a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:07.214 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:07.227+0000 7f5d7766a640 1 -- 172.21.15.135:0/737098803 >> v1:172.21.15.135:6790/0 conn(0x7f5d7013ee90 legacy=0x7f5d7013a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:07.215 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:07.227+0000 7f5d7766a640 1 -- 172.21.15.135:0/737098803 shutdown_connections 2024-03-29T11:27:07.215 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:07.227+0000 7f5d7766a640 1 -- 172.21.15.135:0/737098803 >> 172.21.15.135:0/737098803 conn(0x7f5d700b35e0 msgr2=0x7f5d700ad1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:07.216 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:07.229+0000 7f5d7766a640 1 -- 172.21.15.135:0/737098803 shutdown_connections 2024-03-29T11:27:07.216 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:07.229+0000 7f5d7766a640 1 -- 172.21.15.135:0/737098803 wait complete. 2024-03-29T11:27:07.217 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:07.230+0000 7f5d7766a640 1 librados: shutdown 2024-03-29T11:27:07.235 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:14.042 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:14.195 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.208+0000 7f42db59e640 10 monclient: get_monmap_and_config 2024-03-29T11:27:14.196 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.208+0000 7f42db59e640 10 monclient: build_initial_monmap 2024-03-29T11:27:14.196 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.208+0000 7f42db59e640 10 monclient: monmap: 2024-03-29T11:27:14.196 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:27:14.196 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:14.196 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:27:14.209028+0000 2024-03-29T11:27:14.196 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:27:14.209028+0000 2024-03-29T11:27:14.197 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:14.197 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:14.197 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:27:14.197 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:27:14.197 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:27:14.197 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:14.198 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.210+0000 7f42db59e640 1 Processor -- start 2024-03-29T11:27:14.199 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.210+0000 7f42db59e640 1 -- start start 2024-03-29T11:27:14.199 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.210+0000 7f42db59e640 10 monclient: init 2024-03-29T11:27:14.199 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.211+0000 7f42db59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:14.199 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.211+0000 7f42db59e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:14.200 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.211+0000 7f42db59e640 10 monclient(hunting): picked mon.noname-c con 0x7f42dc0b2800 addr v1:172.21.15.194:6789/0 2024-03-29T11:27:14.200 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.211+0000 7f42db59e640 10 monclient(hunting): picked mon.noname-b con 0x7f42dc142b20 addr v1:172.21.15.135:6790/0 2024-03-29T11:27:14.200 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.211+0000 7f42db59e640 10 monclient(hunting): picked mon.noname-a con 0x7f42dc13c040 addr v1:172.21.15.135:6789/0 2024-03-29T11:27:14.200 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.211+0000 7f42db59e640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42dc0aad10 con 0x7f42dc13c040 2024-03-29T11:27:14.200 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.211+0000 7f42db59e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42dc141f30 con 0x7f42dc0b2800 2024-03-29T11:27:14.200 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.211+0000 7f42db59e640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42dc0aaf60 con 0x7f42dc142b20 2024-03-29T11:27:14.201 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.211+0000 7f42db59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:14.201 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.211+0000 7f42db59e640 10 monclient(hunting): authenticate will time out at 941.580015s 2024-03-29T11:27:14.204 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.216+0000 7f42dad9d640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f42dc13c040 0x7f42dc14d7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:50185/0 (socket says 172.21.15.135:50185) 2024-03-29T11:27:14.204 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.216+0000 7f42dad9d640 1 -- 172.21.15.135:0/3712574504 learned_addr learned my addr 172.21.15.135:0/3712574504 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:27:14.209 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.217+0000 7f42d959a640 1 -- 172.21.15.135:0/3712574504 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804443344 0 0) 0x7f42dc0aad10 con 0x7f42dc13c040 2024-03-29T11:27:14.209 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.217+0000 7f42d959a640 10 monclient(hunting): _negotiate 2024-03-29T11:27:14.209 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.217+0000 7f42d959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:14.209 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.217+0000 7f42d959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:14.209 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.217+0000 7f42d959a640 10 monclient(hunting): my global_id is 4338 2024-03-29T11:27:14.209 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.217+0000 7f42d959a640 1 -- 172.21.15.135:0/3712574504 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42c4003520 con 0x7f42dc13c040 2024-03-29T11:27:14.209 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.217+0000 7f42d959a640 1 -- 172.21.15.135:0/3712574504 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2753186866 0 0) 0x7f42dc141f30 con 0x7f42dc0b2800 2024-03-29T11:27:14.210 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.217+0000 7f42d959a640 10 monclient(hunting): _negotiate 2024-03-29T11:27:14.210 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.217+0000 7f42d959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:14.210 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.217+0000 7f42d959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:14.210 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.217+0000 7f42d959a640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:27:14.210 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.217+0000 7f42d959a640 1 -- 172.21.15.135:0/3712574504 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42dc0aad10 con 0x7f42dc0b2800 2024-03-29T11:27:14.210 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.218+0000 7f42d959a640 1 -- 172.21.15.135:0/3712574504 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4086677532 0 0) 0x7f42c4003520 con 0x7f42dc13c040 2024-03-29T11:27:14.210 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.218+0000 7f42d959a640 1 -- 172.21.15.135:0/3712574504 >> v1:172.21.15.135:6790/0 conn(0x7f42dc142b20 legacy=0x7f42dc13b930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:14.210 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.220+0000 7f42d959a640 1 -- 172.21.15.135:0/3712574504 >> v1:172.21.15.194:6789/0 conn(0x7f42dc0b2800 legacy=0x7f42dc142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:14.210 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.220+0000 7f42d959a640 10 monclient: _finish_hunting 0 2024-03-29T11:27:14.210 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.220+0000 7f42d959a640 1 monclient: found mon.noname-a 2024-03-29T11:27:14.210 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.220+0000 7f42d959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:14.211 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.220+0000 7f42d959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.135:6789/0 2024-03-29T11:27:14.211 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.220+0000 7f42d959a640 1 -- 172.21.15.135:0/3712574504 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42dc0b3900 con 0x7f42dc13c040 2024-03-29T11:27:14.211 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.220+0000 7f42d959a640 10 monclient: _finish_auth 0 2024-03-29T11:27:14.211 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.220+0000 7f42d959a640 10 monclient: _check_auth_tickets 2024-03-29T11:27:14.211 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.220+0000 7f42d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:14.211 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.220+0000 7f42d959a640 1 -- 172.21.15.135:0/3712574504 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f42d0001690 con 0x7f42dc13c040 2024-03-29T11:27:14.211 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.220+0000 7f42d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:14.212 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.220+0000 7f42d959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:14.212 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.220+0000 7f42d959a640 10 monclient: dump: 2024-03-29T11:27:14.212 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:14.212 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:14.212 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:14.212 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:14.212 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:14.212 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:14.212 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:14.212 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:14.212 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:14.212 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:14.214 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.222+0000 7f42db59e640 5 monclient: authenticate success, global_id 4338 2024-03-29T11:27:14.214 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.222+0000 7f42db59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:14.214 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.223+0000 7f42d959a640 1 -- 172.21.15.135:0/3712574504 <== mon.0 v1:172.21.15.135:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42d00018b0 con 0x7f42dc13c040 2024-03-29T11:27:14.214 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.223+0000 7f42d959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:14.214 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.223+0000 7f42d959a640 1 -- 172.21.15.135:0/3712574504 <== mon.0 v1:172.21.15.135:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f42d0001c40 con 0x7f42dc13c040 2024-03-29T11:27:14.214 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.223+0000 7f42d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:14.214 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.223+0000 7f42d959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:14.214 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.223+0000 7f42d959a640 10 monclient: dump: 2024-03-29T11:27:14.214 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:14.214 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:14.214 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:14.214 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:14.215 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:14.215 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:14.215 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:14.215 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:14.215 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:14.215 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:14.215 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.223+0000 7f42db59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:14.215 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.223+0000 7f42db59e640 4 set_mon_vals no callback set 2024-03-29T11:27:14.215 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.224+0000 7f42db59e640 10 monclient: shutdown 2024-03-29T11:27:14.215 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.224+0000 7f42db59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:14.215 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.224+0000 7f42db59e640 1 -- 172.21.15.135:0/3712574504 >> v1:172.21.15.135:6789/0 conn(0x7f42dc13c040 legacy=0x7f42dc14d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:14.215 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.224+0000 7f42db59e640 1 -- 172.21.15.135:0/3712574504 shutdown_connections 2024-03-29T11:27:14.215 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.224+0000 7f42db59e640 1 -- 172.21.15.135:0/3712574504 >> 172.21.15.135:0/3712574504 conn(0x7f42dc06d0e0 msgr2=0x7f42dc066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:14.215 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.224+0000 7f42db59e640 1 -- 172.21.15.135:0/3712574504 shutdown_connections 2024-03-29T11:27:14.216 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.224+0000 7f42db59e640 1 -- 172.21.15.135:0/3712574504 wait complete. 2024-03-29T11:27:14.216 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.225+0000 7f42db59e640 10 monclient: build_initial_monmap 2024-03-29T11:27:14.216 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.225+0000 7f42db59e640 10 monclient: monmap: 2024-03-29T11:27:14.216 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:27:14.216 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:14.216 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:27:14.216 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:27:14.216 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:14.216 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:14.216 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:27:14.216 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:27:14.216 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:27:14.216 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:14.216 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.226+0000 7f42db59e640 1 librados: starting msgr at 2024-03-29T11:27:14.216 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.226+0000 7f42db59e640 1 librados: starting objecter 2024-03-29T11:27:14.217 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.226+0000 7f42db59e640 1 Processor -- start 2024-03-29T11:27:14.219 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.226+0000 7f42db59e640 1 -- start start 2024-03-29T11:27:14.219 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.226+0000 7f42db59e640 1 librados: setting wanted keys 2024-03-29T11:27:14.219 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.226+0000 7f42db59e640 1 librados: calling monclient init 2024-03-29T11:27:14.219 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.226+0000 7f42db59e640 10 monclient: init 2024-03-29T11:27:14.220 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.227+0000 7f42db59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:14.220 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.227+0000 7f42db59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:14.220 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.227+0000 7f42db59e640 10 monclient(hunting): picked mon.noname-b con 0x7f42dc0b2800 addr v1:172.21.15.194:6789/0 2024-03-29T11:27:14.220 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.227+0000 7f42db59e640 10 monclient(hunting): picked mon.noname-a con 0x7f42dc13c040 addr v1:172.21.15.135:6789/0 2024-03-29T11:27:14.220 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.227+0000 7f42db59e640 10 monclient(hunting): picked mon.noname-c con 0x7f42dc142b20 addr v1:172.21.15.135:6790/0 2024-03-29T11:27:14.220 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.227+0000 7f42db59e640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42dc1432a0 con 0x7f42dc13c040 2024-03-29T11:27:14.220 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.227+0000 7f42db59e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42dc1434f0 con 0x7f42dc0b2800 2024-03-29T11:27:14.220 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.227+0000 7f42db59e640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42dc13e620 con 0x7f42dc142b20 2024-03-29T11:27:14.221 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.227+0000 7f42db59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:14.221 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.227+0000 7f42db59e640 10 monclient(hunting): authenticate will time out at 941.595721s 2024-03-29T11:27:14.221 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.232+0000 7f42dad9d640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f42dc142b20 0x7f42dc13ce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:45797/0 (socket says 172.21.15.135:45797) 2024-03-29T11:27:14.221 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.232+0000 7f42dad9d640 1 -- 172.21.15.135:0/1528863310 learned_addr learned my addr 172.21.15.135:0/1528863310 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:27:14.221 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.232+0000 7f42caffd640 10 client.?.objecter ms_handle_connect 0x7f42dc142b20 2024-03-29T11:27:14.221 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.232+0000 7f42caffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:14.222 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.232+0000 7f42caffd640 1 -- 172.21.15.135:0/1528863310 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3791169305 0 0) 0x7f42dc13e620 con 0x7f42dc142b20 2024-03-29T11:27:14.222 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.233+0000 7f42caffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:14.222 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.233+0000 7f42caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:14.222 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.233+0000 7f42caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:14.222 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.233+0000 7f42caffd640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:27:14.222 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.233+0000 7f42caffd640 1 -- 172.21.15.135:0/1528863310 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42bc004eb0 con 0x7f42dc142b20 2024-03-29T11:27:14.233 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.237+0000 7f42caffd640 10 client.?.objecter ms_handle_connect 0x7f42dc0b2800 2024-03-29T11:27:14.233 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.237+0000 7f42caffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:14.233 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.237+0000 7f42caffd640 10 client.?.objecter ms_handle_connect 0x7f42dc13c040 2024-03-29T11:27:14.233 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.237+0000 7f42caffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:14.233 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.238+0000 7f42caffd640 1 -- 172.21.15.135:0/1528863310 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 691547294 0 0) 0x7f42bc004eb0 con 0x7f42dc142b20 2024-03-29T11:27:14.233 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.238+0000 7f42caffd640 1 -- 172.21.15.135:0/1528863310 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42dc13e620 con 0x7f42dc142b20 2024-03-29T11:27:14.234 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.238+0000 7f42caffd640 1 -- 172.21.15.135:0/1528863310 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205911549 0 0) 0x7f42dc1434f0 con 0x7f42dc0b2800 2024-03-29T11:27:14.234 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.238+0000 7f42caffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:14.234 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.238+0000 7f42caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:14.234 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.238+0000 7f42caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:14.234 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.238+0000 7f42caffd640 10 monclient(hunting): my global_id is 4336 2024-03-29T11:27:14.234 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.238+0000 7f42caffd640 1 -- 172.21.15.135:0/1528863310 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42bc004eb0 con 0x7f42dc0b2800 2024-03-29T11:27:14.234 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.238+0000 7f42caffd640 1 -- 172.21.15.135:0/1528863310 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f42d00015d0 con 0x7f42dc142b20 2024-03-29T11:27:14.234 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.238+0000 7f42caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:14.234 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.238+0000 7f42caffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:14.234 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.238+0000 7f42caffd640 10 monclient(hunting): dump: 2024-03-29T11:27:14.234 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:14.234 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:14.235 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:14.235 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:14.235 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:14.235 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:14.235 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:14.235 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:14.235 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:14.235 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:14.235 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.238+0000 7f42caffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:14.235 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.239+0000 7f42caffd640 1 -- 172.21.15.135:0/1528863310 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3005511257 0 0) 0x7f42dc1432a0 con 0x7f42dc13c040 2024-03-29T11:27:14.235 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.239+0000 7f42caffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:14.235 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.239+0000 7f42caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:14.235 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.239+0000 7f42caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:14.235 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.239+0000 7f42caffd640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:27:14.235 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.239+0000 7f42caffd640 1 -- 172.21.15.135:0/1528863310 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42dc1434f0 con 0x7f42dc13c040 2024-03-29T11:27:14.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.239+0000 7f42caffd640 1 -- 172.21.15.135:0/1528863310 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 608771484 0 0) 0x7f42bc004eb0 con 0x7f42dc0b2800 2024-03-29T11:27:14.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.239+0000 7f42caffd640 1 -- 172.21.15.135:0/1528863310 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42dc1432a0 con 0x7f42dc0b2800 2024-03-29T11:27:14.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.239+0000 7f42caffd640 1 -- 172.21.15.135:0/1528863310 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f42d4001a30 con 0x7f42dc0b2800 2024-03-29T11:27:14.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.239+0000 7f42caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:14.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.239+0000 7f42caffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:14.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.239+0000 7f42caffd640 10 monclient(hunting): dump: 2024-03-29T11:27:14.236 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:14.236 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:14.236 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:14.236 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:14.236 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:14.236 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:14.236 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:14.236 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:14.237 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:14.237 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:14.237 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.240+0000 7f42caffd640 1 -- 172.21.15.135:0/1528863310 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3474668767 0 0) 0x7f42dc13e620 con 0x7f42dc142b20 2024-03-29T11:27:14.237 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.240+0000 7f42caffd640 1 -- 172.21.15.135:0/1528863310 >> v1:172.21.15.194:6789/0 conn(0x7f42dc0b2800 legacy=0x7f42dc1d7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:14.237 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.240+0000 7f42caffd640 1 -- 172.21.15.135:0/1528863310 >> v1:172.21.15.135:6789/0 conn(0x7f42dc13c040 legacy=0x7f42dc13c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:14.237 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.240+0000 7f42caffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:14.237 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.240+0000 7f42caffd640 1 monclient: found mon.c 2024-03-29T11:27:14.237 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.240+0000 7f42caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:14.237 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.240+0000 7f42caffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:27:14.237 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.240+0000 7f42caffd640 1 -- 172.21.15.135:0/1528863310 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42dc13e190 con 0x7f42dc142b20 2024-03-29T11:27:14.237 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.240+0000 7f42caffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:14.237 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.240+0000 7f42caffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:14.237 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.240+0000 7f42caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:14.237 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.240+0000 7f42db59e640 5 monclient: authenticate success, global_id 4325 2024-03-29T11:27:14.238 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.240+0000 7f42db59e640 10 monclient: _renew_subs 2024-03-29T11:27:14.238 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.240+0000 7f42db59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:27:14.238 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.240+0000 7f42db59e640 1 -- 172.21.15.135:0/1528863310 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42dc13e930 con 0x7f42dc142b20 2024-03-29T11:27:14.238 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.240+0000 7f42db59e640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:14.238 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.240+0000 7f42db59e640 10 monclient: _renew_subs 2024-03-29T11:27:14.238 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.240+0000 7f42db59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:27:14.238 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.240+0000 7f42db59e640 1 -- 172.21.15.135:0/1528863310 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42dc1390c0 con 0x7f42dc142b20 2024-03-29T11:27:14.238 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.240+0000 7f42db59e640 1 librados: init done 2024-03-29T11:27:14.238 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.241+0000 7f42caffd640 1 -- 172.21.15.135:0/1528863310 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42d0001210 con 0x7f42dc142b20 2024-03-29T11:27:14.238 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.241+0000 7f42caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:14.238 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.241+0000 7f42caffd640 1 -- 172.21.15.135:0/1528863310 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f42d0004c30 con 0x7f42dc142b20 2024-03-29T11:27:14.238 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.241+0000 7f42caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:14.238 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.241+0000 7f42caffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:14.238 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.241+0000 7f42caffd640 10 monclient: dump: 2024-03-29T11:27:14.238 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:14.239 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:14.239 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:14.239 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:14.239 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:14.239 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:14.239 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:14.239 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:14.239 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:14.239 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:14.239 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.242+0000 7f42d8d99640 4 set_mon_vals no callback set 2024-03-29T11:27:14.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.243+0000 7f42db59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:14.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.243+0000 7f42db59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:14.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.244+0000 7f42caffd640 1 -- 172.21.15.135:0/1528863310 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f42d001ac00 con 0x7f42dc142b20 2024-03-29T11:27:14.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.244+0000 7f42caffd640 1 -- 172.21.15.135:0/1528863310 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f42d0084ab0 con 0x7f42dc142b20 2024-03-29T11:27:14.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.244+0000 7f42db59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:27:14.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.245+0000 7f42db59e640 1 -- 172.21.15.135:0/1528863310 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42c0005080 con 0x7f42dc142b20 2024-03-29T11:27:14.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.245+0000 7f42caffd640 10 client.4325.objecter ms_dispatch 0x7f42dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:14.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.245+0000 7f42caffd640 3 client.4325.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:14.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.245+0000 7f42caffd640 3 client.4325.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:14.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.245+0000 7f42caffd640 20 client.4325.objecter dump_active .. 0 homeless 2024-03-29T11:27:14.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.245+0000 7f42da59c640 10 monclient: get_auth_request con 0x7f42bc06e030 auth_method 0 2024-03-29T11:27:14.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.246+0000 7f42caffd640 10 client.4325.objecter ms_handle_connect 0x7f42bc06e030 2024-03-29T11:27:14.241 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.253+0000 7f42caffd640 1 -- 172.21.15.135:0/1528863310 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f42d00501c0 con 0x7f42dc142b20 2024-03-29T11:27:14.241 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.254+0000 7f42caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:14.241 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.254+0000 7f42caffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:14.567 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.580+0000 7f42db59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:14.567 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.580+0000 7f42db59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:14.568 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.580+0000 7f42db59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:27:14.568 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.580+0000 7f42db59e640 1 -- 172.21.15.135:0/1528863310 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42c0005330 con 0x7f42dc142b20 2024-03-29T11:27:14.568 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.581+0000 7f42caffd640 1 -- 172.21.15.135:0/1528863310 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f42d00522c0 con 0x7f42dc142b20 2024-03-29T11:27:14.568 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.581+0000 7f42caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:14.568 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.581+0000 7f42caffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:14.569 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:27:14.569 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:14.572 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.585+0000 7f42db59e640 10 librados: watch_flush enter 2024-03-29T11:27:14.572 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.585+0000 7f42db59e640 10 librados: watch_flush exit 2024-03-29T11:27:14.572 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.585+0000 7f42db59e640 20 client.4325.objecter shutdown clearing up homeless session... 2024-03-29T11:27:14.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.585+0000 7f42db59e640 10 client.4325.objecter successfully canceled tick 2024-03-29T11:27:14.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.585+0000 7f42db59e640 1 -- 172.21.15.135:0/1528863310 >> v1:172.21.15.135:6816/3469875285 conn(0x7f42bc06e030 legacy=0x7f42bc0704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:14.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.585+0000 7f42db59e640 10 monclient: shutdown 2024-03-29T11:27:14.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.585+0000 7f42db59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:14.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.585+0000 7f42db59e640 1 -- 172.21.15.135:0/1528863310 >> v1:172.21.15.135:6790/0 conn(0x7f42dc142b20 legacy=0x7f42dc13ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:14.574 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.586+0000 7f42db59e640 1 -- 172.21.15.135:0/1528863310 shutdown_connections 2024-03-29T11:27:14.574 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.586+0000 7f42db59e640 1 -- 172.21.15.135:0/1528863310 >> 172.21.15.135:0/1528863310 conn(0x7f42dc06d0e0 msgr2=0x7f42dc0b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:14.574 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.586+0000 7f42db59e640 1 -- 172.21.15.135:0/1528863310 shutdown_connections 2024-03-29T11:27:14.574 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.586+0000 7f42db59e640 1 -- 172.21.15.135:0/1528863310 wait complete. 2024-03-29T11:27:14.574 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:14.587+0000 7f42db59e640 1 librados: shutdown 2024-03-29T11:27:14.592 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:21.400 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:21.553 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.564+0000 7ffaee58d640 10 monclient: get_monmap_and_config 2024-03-29T11:27:21.553 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.564+0000 7ffaee58d640 10 monclient: build_initial_monmap 2024-03-29T11:27:21.553 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.564+0000 7ffaee58d640 10 monclient: monmap: 2024-03-29T11:27:21.553 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:27:21.553 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:21.554 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:27:21.566215+0000 2024-03-29T11:27:21.554 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:27:21.566215+0000 2024-03-29T11:27:21.554 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:21.554 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:21.554 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:27:21.554 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:27:21.554 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:27:21.554 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:21.556 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.567+0000 7ffaee58d640 1 Processor -- start 2024-03-29T11:27:21.556 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.567+0000 7ffaee58d640 1 -- start start 2024-03-29T11:27:21.556 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.567+0000 7ffaee58d640 10 monclient: init 2024-03-29T11:27:21.558 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.568+0000 7ffaee58d640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:21.559 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.568+0000 7ffaee58d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:27:21.559 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.568+0000 7ffaee58d640 10 monclient(hunting): picked mon.noname-b con 0x7ffae80b1f60 addr v1:172.21.15.135:6790/0 2024-03-29T11:27:21.559 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.568+0000 7ffaee58d640 10 monclient(hunting): picked mon.noname-c con 0x7ffae8071eb0 addr v1:172.21.15.194:6789/0 2024-03-29T11:27:21.559 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.570+0000 7ffaee58d640 10 monclient(hunting): picked mon.noname-a con 0x7ffae8143d10 addr v1:172.21.15.135:6789/0 2024-03-29T11:27:21.560 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.570+0000 7ffaee58d640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae806ef80 con 0x7ffae8143d10 2024-03-29T11:27:21.560 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.570+0000 7ffaee58d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae806f260 con 0x7ffae8071eb0 2024-03-29T11:27:21.560 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.570+0000 7ffaee58d640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae80b40c0 con 0x7ffae80b1f60 2024-03-29T11:27:21.560 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.570+0000 7ffaee58d640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:21.560 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.570+0000 7ffaee58d640 10 monclient(hunting): authenticate will time out at 948.939477s 2024-03-29T11:27:21.563 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.575+0000 7ffadffff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7ffae8071eb0 0x7ffae8146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:60243/0 (socket says 172.21.15.135:60243) 2024-03-29T11:27:21.563 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.575+0000 7ffadffff640 1 -- 172.21.15.135:0/3960939569 learned_addr learned my addr 172.21.15.135:0/3960939569 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:27:21.572 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.577+0000 7ffae77fe640 1 -- 172.21.15.135:0/3960939569 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 868656218 0 0) 0x7ffae806ef80 con 0x7ffae8143d10 2024-03-29T11:27:21.572 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.577+0000 7ffae77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:21.572 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.577+0000 7ffae77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:21.572 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.577+0000 7ffae77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:21.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.577+0000 7ffae77fe640 10 monclient(hunting): my global_id is 4344 2024-03-29T11:27:21.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.577+0000 7ffae77fe640 1 -- 172.21.15.135:0/3960939569 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad4003520 con 0x7ffae8143d10 2024-03-29T11:27:21.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.577+0000 7ffae77fe640 1 -- 172.21.15.135:0/3960939569 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4074628631 0 0) 0x7ffad4003520 con 0x7ffae8143d10 2024-03-29T11:27:21.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.577+0000 7ffae77fe640 1 -- 172.21.15.135:0/3960939569 >> v1:172.21.15.135:6790/0 conn(0x7ffae80b1f60 legacy=0x7ffae8143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:21.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.577+0000 7ffae77fe640 1 -- 172.21.15.135:0/3960939569 >> v1:172.21.15.194:6789/0 conn(0x7ffae8071eb0 legacy=0x7ffae8146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:21.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.577+0000 7ffae77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:21.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.577+0000 7ffae77fe640 1 monclient: found mon.noname-a 2024-03-29T11:27:21.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.578+0000 7ffae77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:21.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.578+0000 7ffae77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.135:6789/0 2024-03-29T11:27:21.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.578+0000 7ffae77fe640 1 -- 172.21.15.135:0/3960939569 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae8155d70 con 0x7ffae8143d10 2024-03-29T11:27:21.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.578+0000 7ffae77fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:21.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.578+0000 7ffae77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:21.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.578+0000 7ffae77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:21.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.578+0000 7ffae77fe640 1 -- 172.21.15.135:0/3960939569 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7ffae0001a10 con 0x7ffae8143d10 2024-03-29T11:27:21.574 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.578+0000 7ffae77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:21.574 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.578+0000 7ffae77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:21.574 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.578+0000 7ffae77fe640 10 monclient: dump: 2024-03-29T11:27:21.574 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:21.574 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:21.574 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:21.574 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:21.574 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:21.574 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:21.574 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:21.574 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:21.574 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:21.574 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:21.574 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.578+0000 7ffaee58d640 5 monclient: authenticate success, global_id 4344 2024-03-29T11:27:21.575 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.578+0000 7ffaee58d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:21.575 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.578+0000 7ffae77fe640 1 -- 172.21.15.135:0/3960939569 <== mon.0 v1:172.21.15.135:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffae0001ca0 con 0x7ffae8143d10 2024-03-29T11:27:21.575 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.578+0000 7ffae77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:21.575 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.578+0000 7ffae77fe640 1 -- 172.21.15.135:0/3960939569 <== mon.0 v1:172.21.15.135:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7ffae0002030 con 0x7ffae8143d10 2024-03-29T11:27:21.575 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.578+0000 7ffae77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:21.575 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.578+0000 7ffae77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:21.575 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.578+0000 7ffae77fe640 10 monclient: dump: 2024-03-29T11:27:21.575 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:21.575 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:21.575 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:21.575 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:21.575 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:21.575 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:21.575 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:21.576 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:21.576 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:21.576 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:21.576 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.578+0000 7ffaee58d640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:21.576 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.578+0000 7ffaee58d640 4 set_mon_vals no callback set 2024-03-29T11:27:21.576 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.579+0000 7ffaee58d640 10 monclient: shutdown 2024-03-29T11:27:21.576 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.579+0000 7ffaee58d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:21.576 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.579+0000 7ffaee58d640 1 -- 172.21.15.135:0/3960939569 >> v1:172.21.15.135:6789/0 conn(0x7ffae8143d10 legacy=0x7ffae8146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:21.576 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.579+0000 7ffaee58d640 1 -- 172.21.15.135:0/3960939569 shutdown_connections 2024-03-29T11:27:21.576 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.579+0000 7ffaee58d640 1 -- 172.21.15.135:0/3960939569 >> 172.21.15.135:0/3960939569 conn(0x7ffae8066a00 msgr2=0x7ffae8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:21.576 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.579+0000 7ffaee58d640 1 -- 172.21.15.135:0/3960939569 shutdown_connections 2024-03-29T11:27:21.576 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.579+0000 7ffaee58d640 1 -- 172.21.15.135:0/3960939569 wait complete. 2024-03-29T11:27:21.576 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.580+0000 7ffaee58d640 10 monclient: build_initial_monmap 2024-03-29T11:27:21.576 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.580+0000 7ffaee58d640 10 monclient: monmap: 2024-03-29T11:27:21.576 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:27:21.577 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:21.577 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:27:21.577 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:27:21.577 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:21.577 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:21.577 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:27:21.577 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:27:21.577 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:27:21.577 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:21.577 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.580+0000 7ffaee58d640 1 librados: starting msgr at 2024-03-29T11:27:21.577 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.580+0000 7ffaee58d640 1 librados: starting objecter 2024-03-29T11:27:21.577 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.581+0000 7ffaee58d640 1 Processor -- start 2024-03-29T11:27:21.577 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.581+0000 7ffaee58d640 1 -- start start 2024-03-29T11:27:21.577 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.581+0000 7ffaee58d640 1 librados: setting wanted keys 2024-03-29T11:27:21.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.581+0000 7ffaee58d640 1 librados: calling monclient init 2024-03-29T11:27:21.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.581+0000 7ffaee58d640 10 monclient: init 2024-03-29T11:27:21.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.582+0000 7ffaee58d640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:21.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.582+0000 7ffaee58d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:21.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.582+0000 7ffaee58d640 10 monclient(hunting): picked mon.noname-c con 0x7ffae80b1f60 addr v1:172.21.15.135:6790/0 2024-03-29T11:27:21.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.583+0000 7ffaee58d640 10 monclient(hunting): picked mon.noname-a con 0x7ffae8143d10 addr v1:172.21.15.135:6789/0 2024-03-29T11:27:21.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.583+0000 7ffaee58d640 10 monclient(hunting): picked mon.noname-b con 0x7ffae8152030 addr v1:172.21.15.194:6789/0 2024-03-29T11:27:21.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.583+0000 7ffaee58d640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae813ff00 con 0x7ffae8143d10 2024-03-29T11:27:21.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.583+0000 7ffaee58d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae814ba80 con 0x7ffae8152030 2024-03-29T11:27:21.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.583+0000 7ffaee58d640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae813f810 con 0x7ffae80b1f60 2024-03-29T11:27:21.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.583+0000 7ffaee58d640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:21.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.583+0000 7ffaee58d640 10 monclient(hunting): authenticate will time out at 948.952580s 2024-03-29T11:27:21.579 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.585+0000 7ffae7fff640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7ffae80b1f60 0x7ffae8140390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:56463/0 (socket says 172.21.15.135:56463) 2024-03-29T11:27:21.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.585+0000 7ffae7fff640 1 -- 172.21.15.135:0/3593272353 learned_addr learned my addr 172.21.15.135:0/3593272353 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:27:21.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.585+0000 7ffae57fa640 10 client.?.objecter ms_handle_connect 0x7ffae80b1f60 2024-03-29T11:27:21.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.585+0000 7ffae57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:21.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.585+0000 7ffae57fa640 1 -- 172.21.15.135:0/3593272353 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418169411 0 0) 0x7ffae813f810 con 0x7ffae80b1f60 2024-03-29T11:27:21.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.585+0000 7ffae57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:27:21.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.585+0000 7ffae57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:21.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.585+0000 7ffae57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:21.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.585+0000 7ffae57fa640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:27:21.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.586+0000 7ffae57fa640 1 -- 172.21.15.135:0/3593272353 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffac8004eb0 con 0x7ffae80b1f60 2024-03-29T11:27:21.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.586+0000 7ffae57fa640 1 -- 172.21.15.135:0/3593272353 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3637304841 0 0) 0x7ffac8004eb0 con 0x7ffae80b1f60 2024-03-29T11:27:21.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.586+0000 7ffae57fa640 1 -- 172.21.15.135:0/3593272353 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffae813f810 con 0x7ffae80b1f60 2024-03-29T11:27:21.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.586+0000 7ffae57fa640 1 -- 172.21.15.135:0/3593272353 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7ffad8001960 con 0x7ffae80b1f60 2024-03-29T11:27:21.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.586+0000 7ffae57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:21.581 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.586+0000 7ffae57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:21.581 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.586+0000 7ffae57fa640 10 monclient(hunting): dump: 2024-03-29T11:27:21.581 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:21.581 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:21.581 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:21.581 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:21.581 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:21.581 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:21.581 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:21.581 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:21.581 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:21.581 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:21.581 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.586+0000 7ffae57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:21.581 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.590+0000 7ffae57fa640 1 -- 172.21.15.135:0/3593272353 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2522286170 0 0) 0x7ffae813f810 con 0x7ffae80b1f60 2024-03-29T11:27:21.582 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.590+0000 7ffae57fa640 1 -- 172.21.15.135:0/3593272353 >> v1:172.21.15.194:6789/0 conn(0x7ffae8152030 legacy=0x7ffae813f100 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:27:21.582 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.590+0000 7ffae57fa640 1 -- 172.21.15.135:0/3593272353 >> v1:172.21.15.135:6789/0 conn(0x7ffae8143d10 legacy=0x7ffae813e9f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:21.582 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.590+0000 7ffae57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:27:21.582 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.590+0000 7ffae57fa640 1 monclient: found mon.c 2024-03-29T11:27:21.582 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.590+0000 7ffae57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:21.582 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.590+0000 7ffae57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:27:21.582 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.590+0000 7ffae57fa640 1 -- 172.21.15.135:0/3593272353 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae8152ad0 con 0x7ffae80b1f60 2024-03-29T11:27:21.582 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.590+0000 7ffae57fa640 10 monclient: _finish_auth 0 2024-03-29T11:27:21.582 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.590+0000 7ffae57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:27:21.582 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.590+0000 7ffae57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:21.582 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.590+0000 7ffaee58d640 5 monclient: authenticate success, global_id 4328 2024-03-29T11:27:21.582 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.590+0000 7ffaee58d640 10 monclient: _renew_subs 2024-03-29T11:27:21.582 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.590+0000 7ffaee58d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:27:21.583 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.590+0000 7ffaee58d640 1 -- 172.21.15.135:0/3593272353 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffae813fb00 con 0x7ffae80b1f60 2024-03-29T11:27:21.583 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.590+0000 7ffaee58d640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:21.583 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.590+0000 7ffaee58d640 10 monclient: _renew_subs 2024-03-29T11:27:21.583 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.590+0000 7ffaee58d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:27:21.583 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.590+0000 7ffaee58d640 1 -- 172.21.15.135:0/3593272353 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae8154c20 con 0x7ffae80b1f60 2024-03-29T11:27:21.583 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.590+0000 7ffaee58d640 1 librados: init done 2024-03-29T11:27:21.583 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.591+0000 7ffae57fa640 1 -- 172.21.15.135:0/3593272353 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffad8001c80 con 0x7ffae80b1f60 2024-03-29T11:27:21.583 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.591+0000 7ffae57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:21.583 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.591+0000 7ffae57fa640 1 -- 172.21.15.135:0/3593272353 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7ffad8003a40 con 0x7ffae80b1f60 2024-03-29T11:27:21.583 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.591+0000 7ffae57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:21.583 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.591+0000 7ffae57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:21.583 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.591+0000 7ffae57fa640 10 monclient: dump: 2024-03-29T11:27:21.583 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:21.583 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:21.583 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:21.584 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:21.584 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:21.584 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:21.584 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:21.584 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:21.584 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:21.584 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:21.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.591+0000 7ffae6ffd640 4 set_mon_vals no callback set 2024-03-29T11:27:21.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.593+0000 7ffaee58d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:21.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.593+0000 7ffaee58d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:21.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.593+0000 7ffaee58d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:27:21.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.593+0000 7ffaee58d640 1 -- 172.21.15.135:0/3593272353 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffacc005080 con 0x7ffae80b1f60 2024-03-29T11:27:21.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.593+0000 7ffae57fa640 1 -- 172.21.15.135:0/3593272353 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7ffad8003ba0 con 0x7ffae80b1f60 2024-03-29T11:27:21.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.594+0000 7ffae57fa640 1 -- 172.21.15.135:0/3593272353 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7ffad8084af0 con 0x7ffae80b1f60 2024-03-29T11:27:21.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.594+0000 7ffae57fa640 10 client.4328.objecter ms_dispatch 0x7ffae8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:21.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.594+0000 7ffae57fa640 3 client.4328.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:21.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.594+0000 7ffae57fa640 3 client.4328.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:21.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.594+0000 7ffae57fa640 20 client.4328.objecter dump_active .. 0 homeless 2024-03-29T11:27:21.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.600+0000 7ffadffff640 10 monclient: get_auth_request con 0x7ffac806dc30 auth_method 0 2024-03-29T11:27:21.589 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.600+0000 7ffae57fa640 10 client.4328.objecter ms_handle_connect 0x7ffac806dc30 2024-03-29T11:27:21.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.603+0000 7ffae57fa640 1 -- 172.21.15.135:0/3593272353 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffad80503d0 con 0x7ffae80b1f60 2024-03-29T11:27:21.592 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.603+0000 7ffae57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:21.592 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.603+0000 7ffae57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:21.917 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.929+0000 7ffaee58d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:21.918 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.929+0000 7ffaee58d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:21.918 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.929+0000 7ffaee58d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:27:21.918 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.929+0000 7ffaee58d640 1 -- 172.21.15.135:0/3593272353 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffacc005330 con 0x7ffae80b1f60 2024-03-29T11:27:21.919 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.930+0000 7ffae57fa640 1 -- 172.21.15.135:0/3593272353 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7ffad8052300 con 0x7ffae80b1f60 2024-03-29T11:27:21.919 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.930+0000 7ffae57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:21.919 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.930+0000 7ffae57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:21.922 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:27:21.922 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:21.926 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.937+0000 7ffaee58d640 10 librados: watch_flush enter 2024-03-29T11:27:21.926 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.937+0000 7ffaee58d640 10 librados: watch_flush exit 2024-03-29T11:27:21.926 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.937+0000 7ffaee58d640 20 client.4328.objecter shutdown clearing up homeless session... 2024-03-29T11:27:21.926 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.937+0000 7ffaee58d640 10 client.4328.objecter successfully canceled tick 2024-03-29T11:27:21.927 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.938+0000 7ffaee58d640 1 -- 172.21.15.135:0/3593272353 >> v1:172.21.15.135:6816/3469875285 conn(0x7ffac806dc30 legacy=0x7ffac80700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:21.927 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.938+0000 7ffaee58d640 10 monclient: shutdown 2024-03-29T11:27:21.927 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.938+0000 7ffaee58d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:21.927 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.938+0000 7ffaee58d640 1 -- 172.21.15.135:0/3593272353 >> v1:172.21.15.135:6790/0 conn(0x7ffae80b1f60 legacy=0x7ffae8140390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:21.927 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.938+0000 7ffaee58d640 1 -- 172.21.15.135:0/3593272353 shutdown_connections 2024-03-29T11:27:21.927 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.938+0000 7ffaee58d640 1 -- 172.21.15.135:0/3593272353 >> 172.21.15.135:0/3593272353 conn(0x7ffae8066a00 msgr2=0x7ffae8142a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:21.927 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.938+0000 7ffaee58d640 1 -- 172.21.15.135:0/3593272353 shutdown_connections 2024-03-29T11:27:21.927 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.938+0000 7ffaee58d640 1 -- 172.21.15.135:0/3593272353 wait complete. 2024-03-29T11:27:21.927 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:21.938+0000 7ffaee58d640 1 librados: shutdown 2024-03-29T11:27:21.945 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:28.752 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:28.901 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.912+0000 7f7829a16640 10 monclient: get_monmap_and_config 2024-03-29T11:27:28.901 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.912+0000 7f7829a16640 10 monclient: build_initial_monmap 2024-03-29T11:27:28.901 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.912+0000 7f7829a16640 10 monclient: monmap: 2024-03-29T11:27:28.901 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:27:28.902 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:28.902 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:27:28.914410+0000 2024-03-29T11:27:28.902 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:27:28.914410+0000 2024-03-29T11:27:28.902 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:28.902 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:28.902 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:27:28.902 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:27:28.902 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:27:28.902 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:28.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.913+0000 7f7829a16640 1 Processor -- start 2024-03-29T11:27:28.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.913+0000 7f7829a16640 1 -- start start 2024-03-29T11:27:28.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.913+0000 7f7829a16640 10 monclient: init 2024-03-29T11:27:28.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.914+0000 7f7829a16640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:28.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.914+0000 7f7829a16640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:28.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.914+0000 7f7829a16640 10 monclient(hunting): picked mon.noname-b con 0x7f78241457f0 addr v1:172.21.15.135:6790/0 2024-03-29T11:27:28.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.914+0000 7f7829a16640 10 monclient(hunting): picked mon.noname-a con 0x7f782413f650 addr v1:172.21.15.135:6789/0 2024-03-29T11:27:28.907 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.918+0000 7f7822ffd640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f78241457f0 0x7f782413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:46837/0 (socket says 172.21.15.135:46837) 2024-03-29T11:27:28.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.921+0000 7f7829a16640 10 monclient(hunting): picked mon.noname-c con 0x7f78240a9d70 addr v1:172.21.15.194:6789/0 2024-03-29T11:27:28.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.921+0000 7f7829a16640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78240aaef0 con 0x7f782413f650 2024-03-29T11:27:28.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.921+0000 7f7829a16640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7824145bd0 con 0x7f78240a9d70 2024-03-29T11:27:28.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.921+0000 7f7829a16640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7824145eb0 con 0x7f78241457f0 2024-03-29T11:27:28.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.921+0000 7f7829a16640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:28.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.921+0000 7f7829a16640 10 monclient(hunting): authenticate will time out at 956.290532s 2024-03-29T11:27:28.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.921+0000 7f78227fc640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f782413f650 0x7f78240b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:45227/0 (socket says 172.21.15.135:45227) 2024-03-29T11:27:28.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.921+0000 7f78227fc640 1 -- 172.21.15.135:0/4272161270 learned_addr learned my addr 172.21.15.135:0/4272161270 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:27:28.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.925+0000 7f7821ffb640 1 -- 172.21.15.135:0/4272161270 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237726948 0 0) 0x7f7824145eb0 con 0x7f78241457f0 2024-03-29T11:27:28.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.925+0000 7f7821ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:28.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.925+0000 7f7821ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:28.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.925+0000 7f7821ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:28.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.925+0000 7f7821ffb640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:27:28.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.925+0000 7f7821ffb640 1 -- 172.21.15.135:0/4272161270 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f780c003520 con 0x7f78241457f0 2024-03-29T11:27:28.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.925+0000 7f7821ffb640 1 -- 172.21.15.135:0/4272161270 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570354384 0 0) 0x7f78240aaef0 con 0x7f782413f650 2024-03-29T11:27:28.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.925+0000 7f7821ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:28.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.925+0000 7f7821ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:28.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.925+0000 7f7821ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:28.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.925+0000 7f7821ffb640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:27:28.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.925+0000 7f7821ffb640 1 -- 172.21.15.135:0/4272161270 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7824145eb0 con 0x7f782413f650 2024-03-29T11:27:28.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.925+0000 7f7821ffb640 1 -- 172.21.15.135:0/4272161270 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100624393 0 0) 0x7f7824145bd0 con 0x7f78240a9d70 2024-03-29T11:27:28.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.925+0000 7f7821ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:28.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.925+0000 7f7821ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:28.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.925+0000 7f7821ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:28.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.925+0000 7f7821ffb640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:27:28.916 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.925+0000 7f7821ffb640 1 -- 172.21.15.135:0/4272161270 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78240aaef0 con 0x7f78240a9d70 2024-03-29T11:27:28.916 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.926+0000 7f7821ffb640 1 -- 172.21.15.135:0/4272161270 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1076346663 0 0) 0x7f78240aaef0 con 0x7f78240a9d70 2024-03-29T11:27:28.916 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.926+0000 7f7821ffb640 1 -- 172.21.15.135:0/4272161270 >> v1:172.21.15.135:6790/0 conn(0x7f78241457f0 legacy=0x7f782413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:28.916 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.926+0000 7f7821ffb640 1 -- 172.21.15.135:0/4272161270 >> v1:172.21.15.135:6789/0 conn(0x7f782413f650 legacy=0x7f78240b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:28.916 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.926+0000 7f7821ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:28.916 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.926+0000 7f7821ffb640 1 monclient: found mon.noname-c 2024-03-29T11:27:28.916 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.926+0000 7f7821ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:28.916 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.926+0000 7f7821ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:27:28.916 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.926+0000 7f7821ffb640 1 -- 172.21.15.135:0/4272161270 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7824146340 con 0x7f78240a9d70 2024-03-29T11:27:28.916 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.926+0000 7f7821ffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:28.916 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.926+0000 7f7821ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:28.916 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.926+0000 7f7821ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:28.916 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.926+0000 7f7829a16640 5 monclient: authenticate success, global_id 4339 2024-03-29T11:27:28.917 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.926+0000 7f7829a16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:28.917 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.926+0000 7f7821ffb640 1 -- 172.21.15.135:0/4272161270 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f781c0018b0 con 0x7f78240a9d70 2024-03-29T11:27:28.917 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.926+0000 7f7821ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:28.917 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.926+0000 7f7821ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:28.917 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.926+0000 7f7821ffb640 10 monclient: dump: 2024-03-29T11:27:28.917 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:28.917 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:28.917 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:28.917 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:28.917 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:28.917 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:28.917 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:28.918 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:28.918 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:28.918 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:28.918 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.926+0000 7f7829a16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:28.920 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.928+0000 7f7821ffb640 1 -- 172.21.15.135:0/4272161270 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f781c001b00 con 0x7f78240a9d70 2024-03-29T11:27:28.920 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.928+0000 7f7821ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:28.920 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.928+0000 7f7821ffb640 1 -- 172.21.15.135:0/4272161270 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f781c001e90 con 0x7f78240a9d70 2024-03-29T11:27:28.920 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.928+0000 7f7821ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:28.920 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.928+0000 7f7821ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:28.920 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.928+0000 7f7821ffb640 10 monclient: dump: 2024-03-29T11:27:28.920 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:28.920 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:28.920 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:28.921 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:28.921 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:28.921 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:28.921 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:28.921 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:28.921 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:28.921 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:28.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.929+0000 7f7829a16640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:28.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.929+0000 7f7829a16640 4 set_mon_vals no callback set 2024-03-29T11:27:28.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.930+0000 7f7829a16640 10 monclient: shutdown 2024-03-29T11:27:28.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.930+0000 7f7829a16640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:28.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.930+0000 7f7829a16640 1 -- 172.21.15.135:0/4272161270 >> v1:172.21.15.194:6789/0 conn(0x7f78240a9d70 legacy=0x7f782414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:28.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.931+0000 7f7829a16640 1 -- 172.21.15.135:0/4272161270 shutdown_connections 2024-03-29T11:27:28.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.931+0000 7f7829a16640 1 -- 172.21.15.135:0/4272161270 >> 172.21.15.135:0/4272161270 conn(0x7f7824066a00 msgr2=0x7f7824066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:28.923 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.932+0000 7f7829a16640 1 -- 172.21.15.135:0/4272161270 shutdown_connections 2024-03-29T11:27:28.923 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.932+0000 7f7829a16640 1 -- 172.21.15.135:0/4272161270 wait complete. 2024-03-29T11:27:28.923 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.933+0000 7f7829a16640 10 monclient: build_initial_monmap 2024-03-29T11:27:28.923 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.933+0000 7f7829a16640 10 monclient: monmap: 2024-03-29T11:27:28.923 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:27:28.923 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:28.923 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:27:28.923 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:27:28.923 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:28.923 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:28.923 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:27:28.923 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:27:28.924 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:27:28.924 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:28.924 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.933+0000 7f7829a16640 1 librados: starting msgr at 2024-03-29T11:27:28.924 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.933+0000 7f7829a16640 1 librados: starting objecter 2024-03-29T11:27:28.924 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.934+0000 7f7829a16640 1 Processor -- start 2024-03-29T11:27:28.925 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.934+0000 7f7829a16640 1 -- start start 2024-03-29T11:27:28.925 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.934+0000 7f7829a16640 1 librados: setting wanted keys 2024-03-29T11:27:28.925 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.934+0000 7f7829a16640 1 librados: calling monclient init 2024-03-29T11:27:28.925 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.934+0000 7f7829a16640 10 monclient: init 2024-03-29T11:27:28.925 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.935+0000 7f7829a16640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:28.925 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.935+0000 7f7829a16640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:28.928 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.939+0000 7f7829a16640 10 monclient(hunting): picked mon.noname-a con 0x7f78240a9d70 addr v1:172.21.15.135:6789/0 2024-03-29T11:27:28.928 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.939+0000 7f7829a16640 10 monclient(hunting): picked mon.noname-b con 0x7f782413f650 addr v1:172.21.15.194:6789/0 2024-03-29T11:27:28.928 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.940+0000 7f7822ffd640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f78240a9d70 0x7f78241395e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:45423/0 (socket says 172.21.15.135:45423) 2024-03-29T11:27:28.929 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.940+0000 7f7829a16640 10 monclient(hunting): picked mon.noname-c con 0x7f78241457f0 addr v1:172.21.15.135:6790/0 2024-03-29T11:27:28.929 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.940+0000 7f7829a16640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f782406f520 con 0x7f78240a9d70 2024-03-29T11:27:28.930 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.940+0000 7f7829a16640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f782406f800 con 0x7f782413f650 2024-03-29T11:27:28.930 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.940+0000 7f7829a16640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f782413ba40 con 0x7f78241457f0 2024-03-29T11:27:28.930 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.940+0000 7f7829a16640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:28.930 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.940+0000 7f7829a16640 10 monclient(hunting): authenticate will time out at 956.309974s 2024-03-29T11:27:28.930 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.941+0000 7f7822ffd640 1 -- 172.21.15.135:0/156527022 learned_addr learned my addr 172.21.15.135:0/156527022 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:27:28.930 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.941+0000 7f780bfff640 10 client.?.objecter ms_handle_connect 0x7f78240a9d70 2024-03-29T11:27:28.930 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.941+0000 7f780bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:28.930 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.941+0000 7f780bfff640 1 -- 172.21.15.135:0/156527022 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265199352 0 0) 0x7f782406f520 con 0x7f78240a9d70 2024-03-29T11:27:28.930 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.941+0000 7f780bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:28.930 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.941+0000 7f780bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:28.930 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.941+0000 7f780bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:28.930 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.941+0000 7f780bfff640 10 monclient(hunting): my global_id is 4350 2024-03-29T11:27:28.931 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.941+0000 7f780bfff640 1 -- 172.21.15.135:0/156527022 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7810004eb0 con 0x7f78240a9d70 2024-03-29T11:27:28.931 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.941+0000 7f780bfff640 1 -- 172.21.15.135:0/156527022 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2658522192 0 0) 0x7f7810004eb0 con 0x7f78240a9d70 2024-03-29T11:27:28.931 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.941+0000 7f780bfff640 1 -- 172.21.15.135:0/156527022 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f782406f520 con 0x7f78240a9d70 2024-03-29T11:27:28.931 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.942+0000 7f780bfff640 1 -- 172.21.15.135:0/156527022 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f7814001970 con 0x7f78240a9d70 2024-03-29T11:27:28.932 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.942+0000 7f780bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:28.932 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.942+0000 7f780bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:28.932 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.942+0000 7f780bfff640 10 monclient(hunting): dump: 2024-03-29T11:27:28.932 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:28.932 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:28.932 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:28.932 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:28.932 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:28.932 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:28.933 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:28.933 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:28.933 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:28.933 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:28.933 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.942+0000 7f780bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:28.933 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.942+0000 7f780bfff640 1 -- 172.21.15.135:0/156527022 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3399294017 0 0) 0x7f782406f520 con 0x7f78240a9d70 2024-03-29T11:27:28.933 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.942+0000 7f780bfff640 1 -- 172.21.15.135:0/156527022 >> v1:172.21.15.135:6790/0 conn(0x7f78241457f0 legacy=0x7f78241ee840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:28.938 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.946+0000 7f780bfff640 1 -- 172.21.15.135:0/156527022 >> v1:172.21.15.194:6789/0 conn(0x7f782413f650 legacy=0x7f7824139cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:28.938 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.947+0000 7f780bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:28.938 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.947+0000 7f780bfff640 1 monclient: found mon.a 2024-03-29T11:27:28.938 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.947+0000 7f780bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:28.938 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.947+0000 7f780bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:27:28.938 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.947+0000 7f780bfff640 1 -- 172.21.15.135:0/156527022 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f782413bbd0 con 0x7f78240a9d70 2024-03-29T11:27:28.938 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.947+0000 7f780bfff640 10 monclient: _finish_auth 0 2024-03-29T11:27:28.938 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.947+0000 7f780bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:28.938 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.947+0000 7f780bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:28.941 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.947+0000 7f780bfff640 10 client.?.objecter ms_handle_connect 0x7f782413f650 2024-03-29T11:27:28.941 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.947+0000 7f780bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:28.942 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.947+0000 7f7829a16640 5 monclient: authenticate success, global_id 4350 2024-03-29T11:27:28.942 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.947+0000 7f7829a16640 10 monclient: _renew_subs 2024-03-29T11:27:28.942 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.947+0000 7f7829a16640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:27:28.942 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.947+0000 7f7829a16640 1 -- 172.21.15.135:0/156527022 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f782406de90 con 0x7f78240a9d70 2024-03-29T11:27:28.942 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.947+0000 7f7829a16640 10 client.4350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:28.942 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.947+0000 7f7829a16640 10 monclient: _renew_subs 2024-03-29T11:27:28.942 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.947+0000 7f7829a16640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:27:28.943 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.947+0000 7f7829a16640 1 -- 172.21.15.135:0/156527022 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f782406e3c0 con 0x7f78240a9d70 2024-03-29T11:27:28.943 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.948+0000 7f7829a16640 1 librados: init done 2024-03-29T11:27:28.943 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.949+0000 7f780bfff640 1 -- 172.21.15.135:0/156527022 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7814001c70 con 0x7f78240a9d70 2024-03-29T11:27:28.943 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.949+0000 7f780bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:28.943 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.949+0000 7f780bfff640 1 -- 172.21.15.135:0/156527022 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f7814004c30 con 0x7f78240a9d70 2024-03-29T11:27:28.943 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.949+0000 7f780bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:28.943 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.949+0000 7f780bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:28.945 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.949+0000 7f780bfff640 10 monclient: dump: 2024-03-29T11:27:28.945 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:28.945 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:28.945 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:28.945 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:28.945 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:28.945 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:28.945 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:28.946 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:28.946 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:28.946 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:28.946 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.949+0000 7f7820ff9640 4 set_mon_vals no callback set 2024-03-29T11:27:28.946 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.949+0000 7f7829a16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:28.946 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.950+0000 7f7829a16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:28.946 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.950+0000 7f7829a16640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:27:28.946 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.950+0000 7f7829a16640 1 -- 172.21.15.135:0/156527022 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7804005080 con 0x7f78240a9d70 2024-03-29T11:27:28.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.951+0000 7f780bfff640 1 -- 172.21.15.135:0/156527022 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f781401ac00 con 0x7f78240a9d70 2024-03-29T11:27:28.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.951+0000 7f780bfff640 1 -- 172.21.15.135:0/156527022 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f7814084b30 con 0x7f78240a9d70 2024-03-29T11:27:28.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.951+0000 7f780bfff640 10 client.4350.objecter ms_dispatch 0x7f7824000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:28.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.951+0000 7f780bfff640 3 client.4350.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:28.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.951+0000 7f780bfff640 3 client.4350.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:28.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.951+0000 7f780bfff640 20 client.4350.objecter dump_active .. 0 homeless 2024-03-29T11:27:28.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.952+0000 7f78227fc640 10 monclient: get_auth_request con 0x7f781006dc10 auth_method 0 2024-03-29T11:27:28.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.952+0000 7f780bfff640 10 client.4350.objecter ms_handle_connect 0x7f781006dc10 2024-03-29T11:27:28.950 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.961+0000 7f780bfff640 1 -- 172.21.15.135:0/156527022 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7814050410 con 0x7f78240a9d70 2024-03-29T11:27:28.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.961+0000 7f780bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:28.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:28.961+0000 7f780bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:29.269 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:29.280+0000 7f7829a16640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:29.269 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:29.280+0000 7f7829a16640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:29.269 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:29.280+0000 7f7829a16640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:27:29.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:29.280+0000 7f7829a16640 1 -- 172.21.15.135:0/156527022 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7804005330 con 0x7f78240a9d70 2024-03-29T11:27:29.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:29.284+0000 7f780bfff640 1 -- 172.21.15.135:0/156527022 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f7814052340 con 0x7f78240a9d70 2024-03-29T11:27:29.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:29.284+0000 7f780bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:29.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:29.284+0000 7f780bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:29.276 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:27:29.276 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:29.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:29.290+0000 7f7829a16640 10 librados: watch_flush enter 2024-03-29T11:27:29.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:29.290+0000 7f7829a16640 10 librados: watch_flush exit 2024-03-29T11:27:29.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:29.290+0000 7f7829a16640 20 client.4350.objecter shutdown clearing up homeless session... 2024-03-29T11:27:29.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:29.290+0000 7f7829a16640 10 client.4350.objecter successfully canceled tick 2024-03-29T11:27:29.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:29.290+0000 7f7829a16640 1 -- 172.21.15.135:0/156527022 >> v1:172.21.15.135:6816/3469875285 conn(0x7f781006dc10 legacy=0x7f78100700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:29.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:29.290+0000 7f7829a16640 10 monclient: shutdown 2024-03-29T11:27:29.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:29.290+0000 7f7829a16640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:29.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:29.290+0000 7f7829a16640 1 -- 172.21.15.135:0/156527022 >> v1:172.21.15.135:6789/0 conn(0x7f78240a9d70 legacy=0x7f78241395e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:29.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:29.290+0000 7f7829a16640 1 -- 172.21.15.135:0/156527022 shutdown_connections 2024-03-29T11:27:29.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:29.290+0000 7f7829a16640 1 -- 172.21.15.135:0/156527022 >> 172.21.15.135:0/156527022 conn(0x7f7824066a00 msgr2=0x7f78240b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:29.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:29.292+0000 7f7829a16640 1 -- 172.21.15.135:0/156527022 shutdown_connections 2024-03-29T11:27:29.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:29.292+0000 7f7829a16640 1 -- 172.21.15.135:0/156527022 wait complete. 2024-03-29T11:27:29.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:29.292+0000 7f7829a16640 1 librados: shutdown 2024-03-29T11:27:29.303 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:36.110 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:36.262 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.272+0000 7f55cec25640 10 monclient: get_monmap_and_config 2024-03-29T11:27:36.262 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.272+0000 7f55cec25640 10 monclient: build_initial_monmap 2024-03-29T11:27:36.262 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.272+0000 7f55cec25640 10 monclient: monmap: 2024-03-29T11:27:36.262 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:27:36.262 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:36.263 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:27:36.274071+0000 2024-03-29T11:27:36.263 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:27:36.274071+0000 2024-03-29T11:27:36.263 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:36.263 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:36.263 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:27:36.263 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:27:36.263 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:27:36.263 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:36.266 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.274+0000 7f55cec25640 1 Processor -- start 2024-03-29T11:27:36.266 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.277+0000 7f55cec25640 1 -- start start 2024-03-29T11:27:36.266 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.277+0000 7f55cec25640 10 monclient: init 2024-03-29T11:27:36.269 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.278+0000 7f55cec25640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:36.269 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.278+0000 7f55cec25640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:36.269 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.278+0000 7f55cec25640 10 monclient(hunting): picked mon.noname-c con 0x7f55c80b4100 addr v1:172.21.15.194:6789/0 2024-03-29T11:27:36.269 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.278+0000 7f55cec25640 10 monclient(hunting): picked mon.noname-a con 0x7f55c813ee90 addr v1:172.21.15.135:6789/0 2024-03-29T11:27:36.269 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.278+0000 7f55cec25640 10 monclient(hunting): picked mon.noname-b con 0x7f55c80af900 addr v1:172.21.15.135:6790/0 2024-03-29T11:27:36.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.278+0000 7f55cec25640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c80afd10 con 0x7f55c813ee90 2024-03-29T11:27:36.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.278+0000 7f55cec25640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c80af6f0 con 0x7f55c80b4100 2024-03-29T11:27:36.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.278+0000 7f55cec25640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c80aaf60 con 0x7f55c80af900 2024-03-29T11:27:36.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.278+0000 7f55cec25640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:36.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.278+0000 7f55cec25640 10 monclient(hunting): authenticate will time out at 963.647776s 2024-03-29T11:27:36.271 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.282+0000 7f55cd1c2640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f55c80af900 0x7f55c813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:39391/0 (socket says 172.21.15.135:39391) 2024-03-29T11:27:36.271 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.282+0000 7f55cd1c2640 1 -- 172.21.15.135:0/3971018978 learned_addr learned my addr 172.21.15.135:0/3971018978 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:27:36.279 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.287+0000 7f55c77fe640 1 -- 172.21.15.135:0/3971018978 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 401177451 0 0) 0x7f55c80af6f0 con 0x7f55c80b4100 2024-03-29T11:27:36.279 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.287+0000 7f55c77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:36.279 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.287+0000 7f55c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:36.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.287+0000 7f55c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:36.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.287+0000 7f55c77fe640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:27:36.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.287+0000 7f55c77fe640 1 -- 172.21.15.135:0/3971018978 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55b0003520 con 0x7f55c80b4100 2024-03-29T11:27:36.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.287+0000 7f55c77fe640 1 -- 172.21.15.135:0/3971018978 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652467557 0 0) 0x7f55c80aaf60 con 0x7f55c80af900 2024-03-29T11:27:36.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.287+0000 7f55c77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:36.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.287+0000 7f55c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:36.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.287+0000 7f55c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:36.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.287+0000 7f55c77fe640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:27:36.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.287+0000 7f55c77fe640 1 -- 172.21.15.135:0/3971018978 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c80af6f0 con 0x7f55c80af900 2024-03-29T11:27:36.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.287+0000 7f55c77fe640 1 -- 172.21.15.135:0/3971018978 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325331119 0 0) 0x7f55c80afd10 con 0x7f55c813ee90 2024-03-29T11:27:36.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.287+0000 7f55c77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:36.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.287+0000 7f55c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:36.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.287+0000 7f55c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:36.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.287+0000 7f55c77fe640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:27:36.281 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.287+0000 7f55c77fe640 1 -- 172.21.15.135:0/3971018978 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c80aaf60 con 0x7f55c813ee90 2024-03-29T11:27:36.281 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.287+0000 7f55c77fe640 1 -- 172.21.15.135:0/3971018978 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3566457312 0 0) 0x7f55c80af6f0 con 0x7f55c80af900 2024-03-29T11:27:36.281 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.287+0000 7f55c77fe640 1 -- 172.21.15.135:0/3971018978 >> v1:172.21.15.194:6789/0 conn(0x7f55c80b4100 legacy=0x7f55c80a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:36.281 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.288+0000 7f55c77fe640 1 -- 172.21.15.135:0/3971018978 >> v1:172.21.15.135:6789/0 conn(0x7f55c813ee90 legacy=0x7f55c80adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:36.281 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.288+0000 7f55c77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:36.281 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.288+0000 7f55c77fe640 1 monclient: found mon.noname-b 2024-03-29T11:27:36.281 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.288+0000 7f55c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:36.281 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.288+0000 7f55c77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:27:36.281 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.288+0000 7f55c77fe640 1 -- 172.21.15.135:0/3971018978 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c813e970 con 0x7f55c80af900 2024-03-29T11:27:36.281 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.288+0000 7f55c77fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:36.281 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.288+0000 7f55c77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:36.281 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.288+0000 7f55c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:36.281 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.288+0000 7f55c77fe640 1 -- 172.21.15.135:0/3971018978 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f55c00019f0 con 0x7f55c80af900 2024-03-29T11:27:36.281 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.288+0000 7f55c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:36.281 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.288+0000 7f55c77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:36.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.288+0000 7f55c77fe640 10 monclient: dump: 2024-03-29T11:27:36.282 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:36.282 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:36.282 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:36.282 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:36.282 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:36.282 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:36.282 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:36.282 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:36.282 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:36.282 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:36.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.288+0000 7f55cec25640 5 monclient: authenticate success, global_id 4334 2024-03-29T11:27:36.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.288+0000 7f55cec25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:36.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.288+0000 7f55c77fe640 1 -- 172.21.15.135:0/3971018978 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55c0001b50 con 0x7f55c80af900 2024-03-29T11:27:36.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.288+0000 7f55c77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:36.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.288+0000 7f55c77fe640 1 -- 172.21.15.135:0/3971018978 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f55c0001fa0 con 0x7f55c80af900 2024-03-29T11:27:36.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.288+0000 7f55c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:36.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.288+0000 7f55c77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:36.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.288+0000 7f55c77fe640 10 monclient: dump: 2024-03-29T11:27:36.283 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:36.283 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:36.283 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:36.283 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:36.283 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:36.283 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:36.283 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:36.283 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:36.283 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:36.283 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:36.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.288+0000 7f55cec25640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:36.284 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.288+0000 7f55cec25640 4 set_mon_vals no callback set 2024-03-29T11:27:36.284 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.289+0000 7f55cec25640 10 monclient: shutdown 2024-03-29T11:27:36.284 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.289+0000 7f55cec25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:36.284 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.289+0000 7f55cec25640 1 -- 172.21.15.135:0/3971018978 >> v1:172.21.15.135:6790/0 conn(0x7f55c80af900 legacy=0x7f55c813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:36.284 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.292+0000 7f55cec25640 1 -- 172.21.15.135:0/3971018978 shutdown_connections 2024-03-29T11:27:36.285 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.292+0000 7f55cec25640 1 -- 172.21.15.135:0/3971018978 >> 172.21.15.135:0/3971018978 conn(0x7f55c80b35e0 msgr2=0x7f55c80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:36.285 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.292+0000 7f55cec25640 1 -- 172.21.15.135:0/3971018978 shutdown_connections 2024-03-29T11:27:36.285 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.292+0000 7f55cec25640 1 -- 172.21.15.135:0/3971018978 wait complete. 2024-03-29T11:27:36.286 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.293+0000 7f55cec25640 10 monclient: build_initial_monmap 2024-03-29T11:27:36.287 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.293+0000 7f55cec25640 10 monclient: monmap: 2024-03-29T11:27:36.287 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:27:36.287 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:36.287 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:27:36.287 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:27:36.287 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:36.287 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:36.287 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:27:36.287 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:27:36.287 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:27:36.287 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:36.287 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.294+0000 7f55cec25640 1 librados: starting msgr at 2024-03-29T11:27:36.287 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.294+0000 7f55cec25640 1 librados: starting objecter 2024-03-29T11:27:36.287 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.294+0000 7f55cec25640 1 Processor -- start 2024-03-29T11:27:36.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.297+0000 7f55cec25640 1 -- start start 2024-03-29T11:27:36.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.297+0000 7f55cec25640 1 librados: setting wanted keys 2024-03-29T11:27:36.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.297+0000 7f55cec25640 1 librados: calling monclient init 2024-03-29T11:27:36.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.297+0000 7f55cec25640 10 monclient: init 2024-03-29T11:27:36.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.298+0000 7f55cec25640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:36.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.298+0000 7f55cec25640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:36.290 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.301+0000 7f55cec25640 10 monclient(hunting): picked mon.noname-b con 0x7f55c80af900 addr v1:172.21.15.194:6789/0 2024-03-29T11:27:36.294 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.305+0000 7f55cc9c1640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f55c80af900 0x7f55c81405d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:45855/0 (socket says 172.21.15.135:45855) 2024-03-29T11:27:36.294 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.305+0000 7f55cc9c1640 1 -- 172.21.15.135:0/2153116551 learned_addr learned my addr 172.21.15.135:0/2153116551 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:27:36.294 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.305+0000 7f55cec25640 10 monclient(hunting): picked mon.noname-c con 0x7f55c80b4100 addr v1:172.21.15.135:6790/0 2024-03-29T11:27:36.294 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.305+0000 7f55c57fa640 10 client.?.objecter ms_handle_connect 0x7f55c80af900 2024-03-29T11:27:36.294 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.305+0000 7f55cec25640 10 monclient(hunting): picked mon.noname-a con 0x7f55c813ee90 addr v1:172.21.15.135:6789/0 2024-03-29T11:27:36.294 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.305+0000 7f55c57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:36.296 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.305+0000 7f55cec25640 1 -- 172.21.15.135:0/2153116551 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c812d220 con 0x7f55c813ee90 2024-03-29T11:27:36.297 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.305+0000 7f55cec25640 1 -- 172.21.15.135:0/2153116551 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c8140ce0 con 0x7f55c80af900 2024-03-29T11:27:36.297 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.305+0000 7f55cec25640 1 -- 172.21.15.135:0/2153116551 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c8140f30 con 0x7f55c80b4100 2024-03-29T11:27:36.297 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.305+0000 7f55cec25640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:36.297 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.305+0000 7f55cec25640 10 monclient(hunting): authenticate will time out at 963.674759s 2024-03-29T11:27:36.302 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.309+0000 7f55c57fa640 10 client.?.objecter ms_handle_connect 0x7f55c80b4100 2024-03-29T11:27:36.302 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.309+0000 7f55c57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:36.302 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.309+0000 7f55c57fa640 1 -- 172.21.15.135:0/2153116551 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837328050 0 0) 0x7f55c8140ce0 con 0x7f55c80af900 2024-03-29T11:27:36.303 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.309+0000 7f55c57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:27:36.303 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.309+0000 7f55c57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:36.303 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.309+0000 7f55c57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:36.303 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.309+0000 7f55c57fa640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:27:36.303 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.309+0000 7f55c57fa640 1 -- 172.21.15.135:0/2153116551 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55b4004eb0 con 0x7f55c80af900 2024-03-29T11:27:36.303 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.309+0000 7f55c57fa640 1 -- 172.21.15.135:0/2153116551 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152001134 0 0) 0x7f55c8140f30 con 0x7f55c80b4100 2024-03-29T11:27:36.303 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.309+0000 7f55c57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:27:36.303 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.309+0000 7f55c57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:36.303 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.309+0000 7f55c57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:36.303 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.309+0000 7f55c57fa640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:27:36.303 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.309+0000 7f55c57fa640 1 -- 172.21.15.135:0/2153116551 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c8140ce0 con 0x7f55c80b4100 2024-03-29T11:27:36.303 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.309+0000 7f55c57fa640 1 -- 172.21.15.135:0/2153116551 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2007912443 0 0) 0x7f55b4004eb0 con 0x7f55c80af900 2024-03-29T11:27:36.303 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.309+0000 7f55c57fa640 1 -- 172.21.15.135:0/2153116551 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55c8140f30 con 0x7f55c80af900 2024-03-29T11:27:36.303 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.309+0000 7f55c57fa640 1 -- 172.21.15.135:0/2153116551 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f55bc001a70 con 0x7f55c80af900 2024-03-29T11:27:36.304 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.309+0000 7f55c57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:36.304 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.309+0000 7f55c57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:36.304 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.309+0000 7f55c57fa640 10 monclient(hunting): dump: 2024-03-29T11:27:36.304 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:36.304 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:36.304 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:36.304 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:36.304 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:36.305 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:36.305 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:36.305 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:36.305 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:36.305 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:36.305 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.309+0000 7f55c57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:36.305 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.309+0000 7f55c57fa640 1 -- 172.21.15.135:0/2153116551 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2597133962 0 0) 0x7f55c8140ce0 con 0x7f55c80b4100 2024-03-29T11:27:36.305 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.309+0000 7f55c57fa640 1 -- 172.21.15.135:0/2153116551 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55b4004eb0 con 0x7f55c80b4100 2024-03-29T11:27:36.305 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.309+0000 7f55c57fa640 1 -- 172.21.15.135:0/2153116551 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f55b8001890 con 0x7f55c80b4100 2024-03-29T11:27:36.305 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.309+0000 7f55c57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:36.305 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.310+0000 7f55c57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:36.305 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.310+0000 7f55c57fa640 10 monclient(hunting): dump: 2024-03-29T11:27:36.305 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:36.305 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:36.305 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:36.306 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:36.306 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:36.306 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:36.306 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:36.306 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:36.306 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:36.306 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:36.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.310+0000 7f55c57fa640 1 -- 172.21.15.135:0/2153116551 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 941511820 0 0) 0x7f55c8140f30 con 0x7f55c80af900 2024-03-29T11:27:36.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.310+0000 7f55c57fa640 1 -- 172.21.15.135:0/2153116551 >> v1:172.21.15.135:6790/0 conn(0x7f55c80b4100 legacy=0x7f55c81e8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:36.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.310+0000 7f55c57fa640 1 -- 172.21.15.135:0/2153116551 >> v1:172.21.15.135:6789/0 conn(0x7f55c813ee90 legacy=0x7f55c81ef350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:36.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.311+0000 7f55c57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:27:36.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.311+0000 7f55c57fa640 1 monclient: found mon.b 2024-03-29T11:27:36.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.311+0000 7f55c57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:36.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.311+0000 7f55c57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:27:36.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.311+0000 7f55c57fa640 1 -- 172.21.15.135:0/2153116551 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c813f950 con 0x7f55c80af900 2024-03-29T11:27:36.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.311+0000 7f55c57fa640 10 monclient: _finish_auth 0 2024-03-29T11:27:36.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.311+0000 7f55c57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:27:36.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.311+0000 7f55c57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:36.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.311+0000 7f55cec25640 5 monclient: authenticate success, global_id 4348 2024-03-29T11:27:36.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.311+0000 7f55cec25640 10 monclient: _renew_subs 2024-03-29T11:27:36.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.311+0000 7f55cec25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:27:36.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.311+0000 7f55cec25640 1 -- 172.21.15.135:0/2153116551 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55c813fba0 con 0x7f55c80af900 2024-03-29T11:27:36.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.311+0000 7f55cec25640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:36.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.311+0000 7f55cec25640 10 monclient: _renew_subs 2024-03-29T11:27:36.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.311+0000 7f55cec25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:27:36.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.311+0000 7f55cec25640 1 -- 172.21.15.135:0/2153116551 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f55c81e8830 con 0x7f55c80af900 2024-03-29T11:27:36.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.311+0000 7f55cec25640 1 librados: init done 2024-03-29T11:27:36.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.311+0000 7f55c57fa640 1 -- 172.21.15.135:0/2153116551 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55bc0035f0 con 0x7f55c80af900 2024-03-29T11:27:36.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.311+0000 7f55c57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:36.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.311+0000 7f55c57fa640 1 -- 172.21.15.135:0/2153116551 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f55bc003a40 con 0x7f55c80af900 2024-03-29T11:27:36.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.311+0000 7f55c57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:36.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.312+0000 7f55c57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:36.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.312+0000 7f55c57fa640 10 monclient: dump: 2024-03-29T11:27:36.308 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:36.308 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:36.308 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:36.308 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:36.308 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:36.308 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:36.308 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:36.308 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:36.308 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:36.309 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:36.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.312+0000 7f55c6ffd640 4 set_mon_vals no callback set 2024-03-29T11:27:36.310 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.313+0000 7f55cec25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:36.310 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.313+0000 7f55cec25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:36.310 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.313+0000 7f55cec25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:27:36.310 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.313+0000 7f55cec25640 1 -- 172.21.15.135:0/2153116551 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55a8005080 con 0x7f55c80af900 2024-03-29T11:27:36.310 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.314+0000 7f55c57fa640 1 -- 172.21.15.135:0/2153116551 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f55bc003ba0 con 0x7f55c80af900 2024-03-29T11:27:36.311 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.315+0000 7f55c57fa640 1 -- 172.21.15.135:0/2153116551 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f55bc084a60 con 0x7f55c80af900 2024-03-29T11:27:36.311 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.315+0000 7f55c57fa640 10 client.4348.objecter ms_dispatch 0x7f55c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:36.311 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.315+0000 7f55c57fa640 3 client.4348.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:36.311 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.315+0000 7f55c57fa640 3 client.4348.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:36.311 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.315+0000 7f55c57fa640 20 client.4348.objecter dump_active .. 0 homeless 2024-03-29T11:27:36.311 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.315+0000 7f55c7fff640 10 monclient: get_auth_request con 0x7f55b406de40 auth_method 0 2024-03-29T11:27:36.311 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.316+0000 7f55c57fa640 10 client.4348.objecter ms_handle_connect 0x7f55b406de40 2024-03-29T11:27:36.312 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.323+0000 7f55c57fa640 1 -- 172.21.15.135:0/2153116551 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f55bc050340 con 0x7f55c80af900 2024-03-29T11:27:36.312 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.323+0000 7f55c57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:36.312 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.323+0000 7f55c57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:36.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.640+0000 7f55cec25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:36.629 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.640+0000 7f55cec25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:36.629 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.640+0000 7f55cec25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:27:36.629 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.640+0000 7f55cec25640 1 -- 172.21.15.135:0/2153116551 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55a8005330 con 0x7f55c80af900 2024-03-29T11:27:36.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.642+0000 7f55c57fa640 1 -- 172.21.15.135:0/2153116551 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f55bc052270 con 0x7f55c80af900 2024-03-29T11:27:36.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.642+0000 7f55c57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:36.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.642+0000 7f55c57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:36.632 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:27:36.632 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:36.637 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.646+0000 7f55cec25640 10 librados: watch_flush enter 2024-03-29T11:27:36.638 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.648+0000 7f55cec25640 10 librados: watch_flush exit 2024-03-29T11:27:36.638 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.648+0000 7f55cec25640 20 client.4348.objecter shutdown clearing up homeless session... 2024-03-29T11:27:36.638 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.648+0000 7f55cec25640 10 client.4348.objecter successfully canceled tick 2024-03-29T11:27:36.638 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.648+0000 7f55cec25640 1 -- 172.21.15.135:0/2153116551 >> v1:172.21.15.135:6816/3469875285 conn(0x7f55b406de40 legacy=0x7f55b4070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:36.638 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.648+0000 7f55cec25640 10 monclient: shutdown 2024-03-29T11:27:36.638 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.648+0000 7f55cec25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:36.638 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.648+0000 7f55cec25640 1 -- 172.21.15.135:0/2153116551 >> v1:172.21.15.194:6789/0 conn(0x7f55c80af900 legacy=0x7f55c81405d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:36.639 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.650+0000 7f55cec25640 1 -- 172.21.15.135:0/2153116551 shutdown_connections 2024-03-29T11:27:36.639 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.650+0000 7f55cec25640 1 -- 172.21.15.135:0/2153116551 >> 172.21.15.135:0/2153116551 conn(0x7f55c80b35e0 msgr2=0x7f55c80b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:36.639 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.650+0000 7f55cec25640 1 -- 172.21.15.135:0/2153116551 shutdown_connections 2024-03-29T11:27:36.639 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.651+0000 7f55cec25640 1 -- 172.21.15.135:0/2153116551 wait complete. 2024-03-29T11:27:36.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:36.651+0000 7f55cec25640 1 librados: shutdown 2024-03-29T11:27:36.658 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:43.466 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:43.626 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.638+0000 7f3c9f63e640 10 monclient: get_monmap_and_config 2024-03-29T11:27:43.626 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.638+0000 7f3c9f63e640 10 monclient: build_initial_monmap 2024-03-29T11:27:43.627 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.638+0000 7f3c9f63e640 10 monclient: monmap: 2024-03-29T11:27:43.627 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:27:43.627 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:43.627 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:27:43.639226+0000 2024-03-29T11:27:43.627 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:27:43.639226+0000 2024-03-29T11:27:43.627 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:43.627 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:43.627 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:27:43.627 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:27:43.627 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:27:43.627 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:43.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.639+0000 7f3c9f63e640 1 Processor -- start 2024-03-29T11:27:43.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.639+0000 7f3c9f63e640 1 -- start start 2024-03-29T11:27:43.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.639+0000 7f3c9f63e640 10 monclient: init 2024-03-29T11:27:43.632 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.640+0000 7f3c9f63e640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:43.632 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.640+0000 7f3c9f63e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:27:43.632 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.640+0000 7f3c9f63e640 10 monclient(hunting): picked mon.noname-b con 0x7f3c981422b0 addr v1:172.21.15.135:6790/0 2024-03-29T11:27:43.632 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.640+0000 7f3c9f63e640 10 monclient(hunting): picked mon.noname-c con 0x7f3c98142690 addr v1:172.21.15.194:6789/0 2024-03-29T11:27:43.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.640+0000 7f3c9f63e640 10 monclient(hunting): picked mon.noname-a con 0x7f3c980b17d0 addr v1:172.21.15.135:6789/0 2024-03-29T11:27:43.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.640+0000 7f3c9f63e640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c98142c80 con 0x7f3c980b17d0 2024-03-29T11:27:43.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.640+0000 7f3c9f63e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c980aae40 con 0x7f3c98142690 2024-03-29T11:27:43.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.640+0000 7f3c9f63e640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c9813fbd0 con 0x7f3c981422b0 2024-03-29T11:27:43.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.640+0000 7f3c9f63e640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:43.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.640+0000 7f3c9f63e640 10 monclient(hunting): authenticate will time out at 971.008810s 2024-03-29T11:27:43.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.641+0000 7f3c9cbd9640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f3c98142690 0x7f3c980b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:52212/0 (socket says 172.21.15.135:52212) 2024-03-29T11:27:43.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.641+0000 7f3c9cbd9640 1 -- 172.21.15.135:0/1727869635 learned_addr learned my addr 172.21.15.135:0/1727869635 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:27:43.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.642+0000 7f3c8ffff640 1 -- 172.21.15.135:0/1727869635 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2964483726 0 0) 0x7f3c980aae40 con 0x7f3c98142690 2024-03-29T11:27:43.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.642+0000 7f3c8ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:43.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.642+0000 7f3c8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:43.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.642+0000 7f3c8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:43.634 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.642+0000 7f3c8ffff640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:27:43.634 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.642+0000 7f3c8ffff640 1 -- 172.21.15.135:0/1727869635 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c8400bd30 con 0x7f3c98142690 2024-03-29T11:27:43.634 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.642+0000 7f3c8ffff640 1 -- 172.21.15.135:0/1727869635 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3574609714 0 0) 0x7f3c8400bd30 con 0x7f3c98142690 2024-03-29T11:27:43.634 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.642+0000 7f3c8ffff640 1 -- 172.21.15.135:0/1727869635 >> v1:172.21.15.135:6790/0 conn(0x7f3c981422b0 legacy=0x7f3c9813f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:43.634 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.644+0000 7f3c8ffff640 1 -- 172.21.15.135:0/1727869635 >> v1:172.21.15.135:6789/0 conn(0x7f3c980b17d0 legacy=0x7f3c980b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:43.634 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.644+0000 7f3c8ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:43.636 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.644+0000 7f3c8ffff640 1 monclient: found mon.noname-c 2024-03-29T11:27:43.636 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.644+0000 7f3c8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:43.636 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.644+0000 7f3c8ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:27:43.636 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.644+0000 7f3c8ffff640 1 -- 172.21.15.135:0/1727869635 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c98151940 con 0x7f3c98142690 2024-03-29T11:27:43.636 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.644+0000 7f3c8ffff640 10 monclient: _finish_auth 0 2024-03-29T11:27:43.637 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.644+0000 7f3c8ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:43.637 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.644+0000 7f3c8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:43.637 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.644+0000 7f3c9f63e640 5 monclient: authenticate success, global_id 4351 2024-03-29T11:27:43.637 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.644+0000 7f3c9f63e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:43.637 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.644+0000 7f3c8ffff640 1 -- 172.21.15.135:0/1727869635 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f3c90001a10 con 0x7f3c98142690 2024-03-29T11:27:43.637 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.644+0000 7f3c8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:43.637 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.644+0000 7f3c8ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:43.637 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.644+0000 7f3c8ffff640 10 monclient: dump: 2024-03-29T11:27:43.637 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:43.637 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:43.637 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:43.637 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:43.637 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:43.637 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:43.638 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:43.638 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:43.638 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:43.638 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:43.638 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.644+0000 7f3c9f63e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:43.638 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.644+0000 7f3c8ffff640 1 -- 172.21.15.135:0/1727869635 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c90001c30 con 0x7f3c98142690 2024-03-29T11:27:43.638 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.644+0000 7f3c8ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:43.638 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.645+0000 7f3c9f63e640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:43.638 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.645+0000 7f3c9f63e640 4 set_mon_vals no callback set 2024-03-29T11:27:43.638 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.645+0000 7f3c9f63e640 10 monclient: shutdown 2024-03-29T11:27:43.638 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.645+0000 7f3c9f63e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:43.638 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.645+0000 7f3c9f63e640 1 -- 172.21.15.135:0/1727869635 >> v1:172.21.15.194:6789/0 conn(0x7f3c98142690 legacy=0x7f3c980b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:43.638 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.646+0000 7f3c9f63e640 1 -- 172.21.15.135:0/1727869635 shutdown_connections 2024-03-29T11:27:43.638 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.646+0000 7f3c9f63e640 1 -- 172.21.15.135:0/1727869635 >> 172.21.15.135:0/1727869635 conn(0x7f3c98066a00 msgr2=0x7f3c98066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:43.639 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.646+0000 7f3c9f63e640 1 -- 172.21.15.135:0/1727869635 shutdown_connections 2024-03-29T11:27:43.639 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.646+0000 7f3c9f63e640 1 -- 172.21.15.135:0/1727869635 wait complete. 2024-03-29T11:27:43.639 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.647+0000 7f3c9f63e640 10 monclient: build_initial_monmap 2024-03-29T11:27:43.639 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.647+0000 7f3c9f63e640 10 monclient: monmap: 2024-03-29T11:27:43.639 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:27:43.639 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:43.639 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:27:43.639 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:27:43.639 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:43.639 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:43.639 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:27:43.639 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:27:43.639 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:27:43.639 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:43.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.647+0000 7f3c9f63e640 1 librados: starting msgr at 2024-03-29T11:27:43.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.647+0000 7f3c9f63e640 1 librados: starting objecter 2024-03-29T11:27:43.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.647+0000 7f3c9f63e640 1 Processor -- start 2024-03-29T11:27:43.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.647+0000 7f3c9f63e640 1 -- start start 2024-03-29T11:27:43.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.647+0000 7f3c9f63e640 1 librados: setting wanted keys 2024-03-29T11:27:43.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.647+0000 7f3c9f63e640 1 librados: calling monclient init 2024-03-29T11:27:43.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.647+0000 7f3c9f63e640 10 monclient: init 2024-03-29T11:27:43.646 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.648+0000 7f3c9f63e640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:43.646 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.648+0000 7f3c9f63e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:43.646 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.648+0000 7f3c9f63e640 10 monclient(hunting): picked mon.noname-c con 0x7f3c980b17d0 addr v1:172.21.15.135:6790/0 2024-03-29T11:27:43.646 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.648+0000 7f3c9f63e640 10 monclient(hunting): picked mon.noname-a con 0x7f3c981422b0 addr v1:172.21.15.135:6789/0 2024-03-29T11:27:43.646 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.648+0000 7f3c9f63e640 10 monclient(hunting): picked mon.noname-b con 0x7f3c9806e480 addr v1:172.21.15.194:6789/0 2024-03-29T11:27:43.646 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.648+0000 7f3c9f63e640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c98071d20 con 0x7f3c981422b0 2024-03-29T11:27:43.646 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.648+0000 7f3c9f63e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c98070240 con 0x7f3c9806e480 2024-03-29T11:27:43.646 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.648+0000 7f3c9f63e640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c98070520 con 0x7f3c980b17d0 2024-03-29T11:27:43.646 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.648+0000 7f3c9f63e640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:43.646 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.648+0000 7f3c9f63e640 10 monclient(hunting): authenticate will time out at 971.016764s 2024-03-29T11:27:43.646 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.651+0000 7f3c9d3da640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f3c980b17d0 0x7f3c98070f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:43309/0 (socket says 172.21.15.135:43309) 2024-03-29T11:27:43.646 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.651+0000 7f3c9d3da640 1 -- 172.21.15.135:0/4195175705 learned_addr learned my addr 172.21.15.135:0/4195175705 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:27:43.646 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.651+0000 7f3c8dffb640 10 client.?.objecter ms_handle_connect 0x7f3c980b17d0 2024-03-29T11:27:43.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.651+0000 7f3c8dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:43.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.651+0000 7f3c8dffb640 10 client.?.objecter ms_handle_connect 0x7f3c981422b0 2024-03-29T11:27:43.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.651+0000 7f3c8dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:43.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.651+0000 7f3c8dffb640 10 client.?.objecter ms_handle_connect 0x7f3c9806e480 2024-03-29T11:27:43.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.651+0000 7f3c8dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:43.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.651+0000 7f3c8dffb640 1 -- 172.21.15.135:0/4195175705 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3525301461 0 0) 0x7f3c98070520 con 0x7f3c980b17d0 2024-03-29T11:27:43.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.651+0000 7f3c8dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:43.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.651+0000 7f3c8dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:43.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.651+0000 7f3c8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:43.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.651+0000 7f3c8dffb640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:27:43.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.651+0000 7f3c8dffb640 1 -- 172.21.15.135:0/4195175705 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c7c004eb0 con 0x7f3c980b17d0 2024-03-29T11:27:43.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.651+0000 7f3c8dffb640 1 -- 172.21.15.135:0/4195175705 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1715566921 0 0) 0x7f3c98071d20 con 0x7f3c981422b0 2024-03-29T11:27:43.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.651+0000 7f3c8dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:43.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.651+0000 7f3c8dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:43.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.651+0000 7f3c8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:43.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.651+0000 7f3c8dffb640 10 monclient(hunting): my global_id is 4356 2024-03-29T11:27:43.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.651+0000 7f3c8dffb640 1 -- 172.21.15.135:0/4195175705 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c98070520 con 0x7f3c981422b0 2024-03-29T11:27:43.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.652+0000 7f3c8dffb640 1 -- 172.21.15.135:0/4195175705 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2586857722 0 0) 0x7f3c98070240 con 0x7f3c9806e480 2024-03-29T11:27:43.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.652+0000 7f3c8dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:43.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.652+0000 7f3c8dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:43.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.652+0000 7f3c8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:43.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.652+0000 7f3c8dffb640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:27:43.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.652+0000 7f3c8dffb640 1 -- 172.21.15.135:0/4195175705 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c98071d20 con 0x7f3c9806e480 2024-03-29T11:27:43.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.652+0000 7f3c8dffb640 1 -- 172.21.15.135:0/4195175705 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2709916294 0 0) 0x7f3c98070520 con 0x7f3c981422b0 2024-03-29T11:27:43.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.652+0000 7f3c8dffb640 1 -- 172.21.15.135:0/4195175705 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c98070240 con 0x7f3c981422b0 2024-03-29T11:27:43.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.652+0000 7f3c8dffb640 1 -- 172.21.15.135:0/4195175705 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 265432432 0 0) 0x7f3c98071d20 con 0x7f3c9806e480 2024-03-29T11:27:43.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.652+0000 7f3c8dffb640 1 -- 172.21.15.135:0/4195175705 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c98070520 con 0x7f3c9806e480 2024-03-29T11:27:43.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.652+0000 7f3c8dffb640 1 -- 172.21.15.135:0/4195175705 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f3c90004b60 con 0x7f3c981422b0 2024-03-29T11:27:43.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.652+0000 7f3c8dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:43.649 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.652+0000 7f3c8dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:43.649 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.652+0000 7f3c8dffb640 10 monclient(hunting): dump: 2024-03-29T11:27:43.649 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:43.649 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:43.649 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:43.649 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:43.649 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:43.649 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:43.649 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:43.649 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:43.649 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:43.649 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:43.649 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.652+0000 7f3c8dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:43.649 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.652+0000 7f3c8dffb640 1 -- 172.21.15.135:0/4195175705 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f3c94001660 con 0x7f3c9806e480 2024-03-29T11:27:43.649 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.652+0000 7f3c8dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:43.650 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.652+0000 7f3c8dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:43.650 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.652+0000 7f3c8dffb640 10 monclient(hunting): dump: 2024-03-29T11:27:43.650 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:43.650 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:43.650 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:43.650 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:43.650 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:43.650 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:43.650 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:43.650 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:43.650 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:43.650 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:43.650 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.653+0000 7f3c8dffb640 1 -- 172.21.15.135:0/4195175705 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3388079676 0 0) 0x7f3c98070520 con 0x7f3c9806e480 2024-03-29T11:27:43.650 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.653+0000 7f3c8dffb640 1 -- 172.21.15.135:0/4195175705 >> v1:172.21.15.135:6790/0 conn(0x7f3c980b17d0 legacy=0x7f3c98070f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:43.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.653+0000 7f3c8dffb640 1 -- 172.21.15.135:0/4195175705 >> v1:172.21.15.135:6789/0 conn(0x7f3c981422b0 legacy=0x7f3c98071610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:43.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.653+0000 7f3c8dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:43.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.653+0000 7f3c8dffb640 1 monclient: found mon.b 2024-03-29T11:27:43.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.653+0000 7f3c8dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:43.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.653+0000 7f3c8dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:27:43.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.653+0000 7f3c8dffb640 1 -- 172.21.15.135:0/4195175705 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c98070800 con 0x7f3c9806e480 2024-03-29T11:27:43.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.653+0000 7f3c8dffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:43.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.653+0000 7f3c8dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:43.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.653+0000 7f3c8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:43.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.653+0000 7f3c8dffb640 1 -- 172.21.15.135:0/4195175705 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c940017c0 con 0x7f3c9806e480 2024-03-29T11:27:43.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.653+0000 7f3c8dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:43.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.653+0000 7f3c8dffb640 1 -- 172.21.15.135:0/4195175705 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f3c940035f0 con 0x7f3c9806e480 2024-03-29T11:27:43.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.653+0000 7f3c8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:43.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.653+0000 7f3c8dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:43.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.653+0000 7f3c8dffb640 10 monclient: dump: 2024-03-29T11:27:43.652 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:43.652 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:43.652 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:43.652 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:43.652 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:43.652 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:43.652 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:43.652 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:43.652 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:43.652 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:43.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.653+0000 7f3c8f7fe640 4 set_mon_vals no callback set 2024-03-29T11:27:43.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.655+0000 7f3c9f63e640 5 monclient: authenticate success, global_id 4354 2024-03-29T11:27:43.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.655+0000 7f3c9f63e640 10 monclient: _renew_subs 2024-03-29T11:27:43.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.655+0000 7f3c9f63e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:27:43.653 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.655+0000 7f3c9f63e640 1 -- 172.21.15.135:0/4195175705 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c9806f120 con 0x7f3c9806e480 2024-03-29T11:27:43.653 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.655+0000 7f3c9f63e640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:43.653 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.655+0000 7f3c9f63e640 10 monclient: _renew_subs 2024-03-29T11:27:43.653 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.655+0000 7f3c9f63e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:27:43.653 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.655+0000 7f3c9f63e640 1 -- 172.21.15.135:0/4195175705 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c981df3d0 con 0x7f3c9806e480 2024-03-29T11:27:43.653 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.655+0000 7f3c9f63e640 1 librados: init done 2024-03-29T11:27:43.653 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.656+0000 7f3c9f63e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:43.653 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.656+0000 7f3c9f63e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:43.653 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.656+0000 7f3c9f63e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:27:43.653 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.656+0000 7f3c9f63e640 1 -- 172.21.15.135:0/4195175705 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c98075f10 con 0x7f3c9806e480 2024-03-29T11:27:43.654 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.657+0000 7f3c8dffb640 1 -- 172.21.15.135:0/4195175705 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f3c940195c0 con 0x7f3c9806e480 2024-03-29T11:27:43.654 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.658+0000 7f3c8dffb640 1 -- 172.21.15.135:0/4195175705 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f3c94083630 con 0x7f3c9806e480 2024-03-29T11:27:43.654 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.658+0000 7f3c8dffb640 10 client.4354.objecter ms_dispatch 0x7f3c98000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:43.654 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.658+0000 7f3c8dffb640 3 client.4354.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:43.654 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.658+0000 7f3c8dffb640 3 client.4354.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:43.654 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.658+0000 7f3c8dffb640 20 client.4354.objecter dump_active .. 0 homeless 2024-03-29T11:27:43.654 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.658+0000 7f3c9d3da640 10 monclient: get_auth_request con 0x7f3c7c06e020 auth_method 0 2024-03-29T11:27:43.655 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.658+0000 7f3c8dffb640 10 client.4354.objecter ms_handle_connect 0x7f3c7c06e020 2024-03-29T11:27:43.655 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.664+0000 7f3c8dffb640 1 -- 172.21.15.135:0/4195175705 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3c9404ed40 con 0x7f3c9806e480 2024-03-29T11:27:43.655 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.664+0000 7f3c8dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:43.655 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.664+0000 7f3c8dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:43.980 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:27:43.980 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:43.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.990+0000 7f3c9f63e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:43.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.990+0000 7f3c9f63e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:43.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.990+0000 7f3c9f63e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:27:43.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.990+0000 7f3c9f63e640 1 -- 172.21.15.135:0/4195175705 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c9804eee0 con 0x7f3c9806e480 2024-03-29T11:27:43.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.991+0000 7f3c8dffb640 1 -- 172.21.15.135:0/4195175705 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f3c94050e40 con 0x7f3c9806e480 2024-03-29T11:27:43.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.991+0000 7f3c8dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:43.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.991+0000 7f3c8dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:43.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.994+0000 7f3c9f63e640 10 librados: watch_flush enter 2024-03-29T11:27:43.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.995+0000 7f3c9f63e640 10 librados: watch_flush exit 2024-03-29T11:27:43.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.995+0000 7f3c9f63e640 20 client.4354.objecter shutdown clearing up homeless session... 2024-03-29T11:27:43.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.995+0000 7f3c9f63e640 10 client.4354.objecter successfully canceled tick 2024-03-29T11:27:43.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.997+0000 7f3c9f63e640 1 -- 172.21.15.135:0/4195175705 >> v1:172.21.15.135:6816/3469875285 conn(0x7f3c7c06e020 legacy=0x7f3c7c0704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:43.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.997+0000 7f3c9f63e640 10 monclient: shutdown 2024-03-29T11:27:43.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.997+0000 7f3c9f63e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:43.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.997+0000 7f3c9f63e640 1 -- 172.21.15.135:0/4195175705 >> v1:172.21.15.194:6789/0 conn(0x7f3c9806e480 legacy=0x7f3c981ddcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:43.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.997+0000 7f3c9f63e640 1 -- 172.21.15.135:0/4195175705 shutdown_connections 2024-03-29T11:27:43.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.997+0000 7f3c9f63e640 1 -- 172.21.15.135:0/4195175705 >> 172.21.15.135:0/4195175705 conn(0x7f3c98066a00 msgr2=0x7f3c980b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:43.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.998+0000 7f3c9f63e640 1 -- 172.21.15.135:0/4195175705 shutdown_connections 2024-03-29T11:27:43.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.998+0000 7f3c9f63e640 1 -- 172.21.15.135:0/4195175705 wait complete. 2024-03-29T11:27:43.987 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:43.998+0000 7f3c9f63e640 1 librados: shutdown 2024-03-29T11:27:44.003 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:50.810 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:50.959 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.970+0000 7f169d3a8640 10 monclient: get_monmap_and_config 2024-03-29T11:27:50.959 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.970+0000 7f169d3a8640 10 monclient: build_initial_monmap 2024-03-29T11:27:50.959 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.970+0000 7f169d3a8640 10 monclient: monmap: 2024-03-29T11:27:50.959 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:27:50.959 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:50.959 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:27:50.971155+0000 2024-03-29T11:27:50.959 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:27:50.971155+0000 2024-03-29T11:27:50.959 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:50.959 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:50.959 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:27:50.960 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:27:50.960 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:27:50.960 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:50.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.971+0000 7f169d3a8640 1 Processor -- start 2024-03-29T11:27:50.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.971+0000 7f169d3a8640 1 -- start start 2024-03-29T11:27:50.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.971+0000 7f169d3a8640 10 monclient: init 2024-03-29T11:27:50.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.972+0000 7f169d3a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:50.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.972+0000 7f169d3a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:50.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.973+0000 7f169d3a8640 10 monclient(hunting): picked mon.noname-b con 0x7f16980b46b0 addr v1:172.21.15.135:6790/0 2024-03-29T11:27:50.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.973+0000 7f169d3a8640 10 monclient(hunting): picked mon.noname-a con 0x7f16980aeae0 addr v1:172.21.15.135:6789/0 2024-03-29T11:27:50.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.973+0000 7f169d3a8640 10 monclient(hunting): picked mon.noname-c con 0x7f169813fa10 addr v1:172.21.15.194:6789/0 2024-03-29T11:27:50.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.973+0000 7f168ffff640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f16980aeae0 0x7f169813f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:53454/0 (socket says 172.21.15.135:53454) 2024-03-29T11:27:50.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.973+0000 7f168ffff640 1 -- 172.21.15.135:0/3195825099 learned_addr learned my addr 172.21.15.135:0/3195825099 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:27:50.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.973+0000 7f169d3a8640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16980b06a0 con 0x7f16980aeae0 2024-03-29T11:27:50.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.973+0000 7f169d3a8640 1 -- 172.21.15.135:0/3195825099 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16980b4230 con 0x7f169813fa10 2024-03-29T11:27:50.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.973+0000 7f169d3a8640 1 -- 172.21.15.135:0/3195825099 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16980b0910 con 0x7f16980b46b0 2024-03-29T11:27:50.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.973+0000 7f169d3a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:50.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.973+0000 7f169d3a8640 10 monclient(hunting): authenticate will time out at 978.341574s 2024-03-29T11:27:50.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.973+0000 7f16967fc640 1 -- 172.21.15.135:0/3195825099 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095996191 0 0) 0x7f16980b06a0 con 0x7f16980aeae0 2024-03-29T11:27:50.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.973+0000 7f16967fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:50.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.973+0000 7f16967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:50.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.973+0000 7f16967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:50.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.973+0000 7f16967fc640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:27:50.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.973+0000 7f16967fc640 1 -- 172.21.15.135:0/3195825099 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1684003520 con 0x7f16980aeae0 2024-03-29T11:27:50.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.974+0000 7f16967fc640 1 -- 172.21.15.135:0/3195825099 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3048838691 0 0) 0x7f1684003520 con 0x7f16980aeae0 2024-03-29T11:27:50.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.974+0000 7f16967fc640 1 -- 172.21.15.135:0/3195825099 >> v1:172.21.15.135:6790/0 conn(0x7f16980b46b0 legacy=0x7f16980b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:50.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.974+0000 7f16967fc640 1 -- 172.21.15.135:0/3195825099 >> v1:172.21.15.194:6789/0 conn(0x7f169813fa10 legacy=0x7f169814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:50.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.974+0000 7f16967fc640 10 monclient: _finish_hunting 0 2024-03-29T11:27:50.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.974+0000 7f16967fc640 1 monclient: found mon.noname-a 2024-03-29T11:27:50.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.974+0000 7f16967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:50.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.974+0000 7f16967fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.135:6789/0 2024-03-29T11:27:50.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.974+0000 7f16967fc640 1 -- 172.21.15.135:0/3195825099 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16980af6d0 con 0x7f16980aeae0 2024-03-29T11:27:50.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.974+0000 7f16967fc640 10 monclient: _finish_auth 0 2024-03-29T11:27:50.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.974+0000 7f16967fc640 10 monclient: _check_auth_tickets 2024-03-29T11:27:50.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.974+0000 7f16967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:50.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.974+0000 7f16967fc640 1 -- 172.21.15.135:0/3195825099 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f1680001690 con 0x7f16980aeae0 2024-03-29T11:27:50.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.974+0000 7f16967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:50.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.974+0000 7f16967fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:50.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.974+0000 7f16967fc640 10 monclient: dump: 2024-03-29T11:27:50.967 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:50.967 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:50.967 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:50.967 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:50.968 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:50.968 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:50.968 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:50.968 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:50.968 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:50.968 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:50.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.974+0000 7f169d3a8640 5 monclient: authenticate success, global_id 4359 2024-03-29T11:27:50.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.974+0000 7f169d3a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:50.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.974+0000 7f16967fc640 1 -- 172.21.15.135:0/3195825099 <== mon.0 v1:172.21.15.135:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1680001860 con 0x7f16980aeae0 2024-03-29T11:27:50.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.974+0000 7f16967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:50.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.974+0000 7f16967fc640 1 -- 172.21.15.135:0/3195825099 <== mon.0 v1:172.21.15.135:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f1680001c70 con 0x7f16980aeae0 2024-03-29T11:27:50.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.974+0000 7f16967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:50.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.974+0000 7f16967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:50.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.974+0000 7f16967fc640 10 monclient: dump: 2024-03-29T11:27:50.968 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:50.969 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:50.969 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:50.969 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:50.969 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:50.969 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:50.969 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:50.969 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:50.969 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:50.969 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:50.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.975+0000 7f169d3a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:50.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.975+0000 7f169d3a8640 4 set_mon_vals no callback set 2024-03-29T11:27:50.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.975+0000 7f169d3a8640 10 monclient: shutdown 2024-03-29T11:27:50.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.975+0000 7f169d3a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:50.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.975+0000 7f169d3a8640 1 -- 172.21.15.135:0/3195825099 >> v1:172.21.15.135:6789/0 conn(0x7f16980aeae0 legacy=0x7f169813f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:50.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.976+0000 7f169d3a8640 1 -- 172.21.15.135:0/3195825099 shutdown_connections 2024-03-29T11:27:50.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.976+0000 7f169d3a8640 1 -- 172.21.15.135:0/3195825099 >> 172.21.15.135:0/3195825099 conn(0x7f1698066a00 msgr2=0x7f1698066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:50.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.976+0000 7f169d3a8640 1 -- 172.21.15.135:0/3195825099 shutdown_connections 2024-03-29T11:27:50.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.976+0000 7f169d3a8640 1 -- 172.21.15.135:0/3195825099 wait complete. 2024-03-29T11:27:50.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.977+0000 7f169d3a8640 10 monclient: build_initial_monmap 2024-03-29T11:27:50.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.977+0000 7f169d3a8640 10 monclient: monmap: 2024-03-29T11:27:50.977 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:27:50.977 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:50.977 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:27:50.977 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:27:50.977 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:50.977 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:50.977 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:27:50.977 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:27:50.977 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:27:50.978 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:50.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.977+0000 7f169d3a8640 1 librados: starting msgr at 2024-03-29T11:27:50.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.977+0000 7f169d3a8640 1 librados: starting objecter 2024-03-29T11:27:50.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.978+0000 7f169d3a8640 1 Processor -- start 2024-03-29T11:27:50.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.978+0000 7f169d3a8640 1 -- start start 2024-03-29T11:27:50.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.978+0000 7f169d3a8640 1 librados: setting wanted keys 2024-03-29T11:27:50.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.978+0000 7f169d3a8640 1 librados: calling monclient init 2024-03-29T11:27:50.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.978+0000 7f169d3a8640 10 monclient: init 2024-03-29T11:27:50.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.979+0000 7f169d3a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:50.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.979+0000 7f169d3a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:50.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.979+0000 7f169d3a8640 10 monclient(hunting): picked mon.noname-b con 0x7f16980aeae0 addr v1:172.21.15.194:6789/0 2024-03-29T11:27:50.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.979+0000 7f169d3a8640 10 monclient(hunting): picked mon.noname-a con 0x7f16980b46b0 addr v1:172.21.15.135:6789/0 2024-03-29T11:27:50.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.979+0000 7f169d3a8640 10 monclient(hunting): picked mon.noname-c con 0x7f169813fa10 addr v1:172.21.15.135:6790/0 2024-03-29T11:27:50.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.979+0000 7f169d3a8640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16981d9830 con 0x7f16980b46b0 2024-03-29T11:27:50.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.979+0000 7f169d3a8640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1698145b60 con 0x7f16980aeae0 2024-03-29T11:27:50.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.979+0000 7f169d3a8640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1698144a60 con 0x7f169813fa10 2024-03-29T11:27:50.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.979+0000 7f169d3a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:50.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.979+0000 7f169d3a8640 10 monclient(hunting): authenticate will time out at 978.347403s 2024-03-29T11:27:50.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.979+0000 7f1696ffd640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f16980aeae0 0x7f16981dbd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:45624/0 (socket says 172.21.15.135:45624) 2024-03-29T11:27:50.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.979+0000 7f1696ffd640 1 -- 172.21.15.135:0/734553213 learned_addr learned my addr 172.21.15.135:0/734553213 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:27:50.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.979+0000 7f168f7fe640 10 client.?.objecter ms_handle_connect 0x7f16980aeae0 2024-03-29T11:27:50.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.979+0000 7f168f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:50.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.979+0000 7f168f7fe640 10 client.?.objecter ms_handle_connect 0x7f16980b46b0 2024-03-29T11:27:50.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.979+0000 7f168f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:50.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.979+0000 7f168f7fe640 10 client.?.objecter ms_handle_connect 0x7f169813fa10 2024-03-29T11:27:50.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.980+0000 7f168f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:50.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.980+0000 7f168f7fe640 1 -- 172.21.15.135:0/734553213 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285369581 0 0) 0x7f16981d9830 con 0x7f16980b46b0 2024-03-29T11:27:50.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.980+0000 7f168f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:50.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.980+0000 7f168f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:50.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.980+0000 7f168f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:50.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.980+0000 7f168f7fe640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:27:50.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.980+0000 7f168f7fe640 1 -- 172.21.15.135:0/734553213 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1678004eb0 con 0x7f16980b46b0 2024-03-29T11:27:50.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.980+0000 7f168f7fe640 1 -- 172.21.15.135:0/734553213 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175863670 0 0) 0x7f1698145b60 con 0x7f16980aeae0 2024-03-29T11:27:50.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.980+0000 7f168f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:50.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.980+0000 7f168f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:50.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.980+0000 7f168f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:50.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.980+0000 7f168f7fe640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:27:50.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.980+0000 7f168f7fe640 1 -- 172.21.15.135:0/734553213 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16981d9830 con 0x7f16980aeae0 2024-03-29T11:27:50.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.980+0000 7f168f7fe640 1 -- 172.21.15.135:0/734553213 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929766586 0 0) 0x7f1698144a60 con 0x7f169813fa10 2024-03-29T11:27:50.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.980+0000 7f168f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:50.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.980+0000 7f168f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:50.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.980+0000 7f168f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:50.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.980+0000 7f168f7fe640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:27:50.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.980+0000 7f168f7fe640 1 -- 172.21.15.135:0/734553213 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1698145b60 con 0x7f169813fa10 2024-03-29T11:27:50.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.980+0000 7f168f7fe640 1 -- 172.21.15.135:0/734553213 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4171019744 0 0) 0x7f1678004eb0 con 0x7f16980b46b0 2024-03-29T11:27:50.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.980+0000 7f168f7fe640 1 -- 172.21.15.135:0/734553213 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1698144a60 con 0x7f16980b46b0 2024-03-29T11:27:50.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.980+0000 7f168f7fe640 1 -- 172.21.15.135:0/734553213 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f1680001030 con 0x7f16980b46b0 2024-03-29T11:27:50.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.980+0000 7f168f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:50.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.980+0000 7f168f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:50.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.980+0000 7f168f7fe640 10 monclient(hunting): dump: 2024-03-29T11:27:50.981 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:50.982 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:50.982 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:50.982 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:50.982 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:50.982 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:50.982 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:50.982 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:50.982 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:50.982 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:50.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.980+0000 7f168f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:50.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f168f7fe640 1 -- 172.21.15.135:0/734553213 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2839734634 0 0) 0x7f16981d9830 con 0x7f16980aeae0 2024-03-29T11:27:50.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f168f7fe640 1 -- 172.21.15.135:0/734553213 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1678004eb0 con 0x7f16980aeae0 2024-03-29T11:27:50.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f168f7fe640 1 -- 172.21.15.135:0/734553213 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f16880019b0 con 0x7f16980aeae0 2024-03-29T11:27:50.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f168f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:50.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f168f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:50.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f168f7fe640 10 monclient(hunting): dump: 2024-03-29T11:27:50.983 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:50.983 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:50.983 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:50.983 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:50.983 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:50.983 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:50.983 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:50.983 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:50.983 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:50.983 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:50.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f168f7fe640 1 -- 172.21.15.135:0/734553213 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 158313242 0 0) 0x7f1698145b60 con 0x7f169813fa10 2024-03-29T11:27:50.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f168f7fe640 1 -- 172.21.15.135:0/734553213 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16981d9830 con 0x7f169813fa10 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f168f7fe640 1 -- 172.21.15.135:0/734553213 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f1690001a10 con 0x7f169813fa10 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f168f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f168f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f168f7fe640 10 monclient(hunting): dump: 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:50.984 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:50.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f168f7fe640 1 -- 172.21.15.135:0/734553213 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3436631434 0 0) 0x7f1678004eb0 con 0x7f16980aeae0 2024-03-29T11:27:50.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f168f7fe640 1 -- 172.21.15.135:0/734553213 >> v1:172.21.15.135:6790/0 conn(0x7f169813fa10 legacy=0x7f16981d9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:50.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f168f7fe640 1 -- 172.21.15.135:0/734553213 >> v1:172.21.15.135:6789/0 conn(0x7f16980b46b0 legacy=0x7f16981d8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:50.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f168f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:50.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f168f7fe640 1 monclient: found mon.b 2024-03-29T11:27:50.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f168f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:50.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f168f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:27:50.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f168f7fe640 1 -- 172.21.15.135:0/734553213 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1698144bf0 con 0x7f16980aeae0 2024-03-29T11:27:50.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f168f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:50.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f168f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:50.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f168f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:50.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f169d3a8640 5 monclient: authenticate success, global_id 4357 2024-03-29T11:27:50.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f169d3a8640 10 monclient: _renew_subs 2024-03-29T11:27:50.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f169d3a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:27:50.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f169d3a8640 1 -- 172.21.15.135:0/734553213 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16980b6900 con 0x7f16980aeae0 2024-03-29T11:27:50.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f169d3a8640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:50.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f169d3a8640 10 monclient: _renew_subs 2024-03-29T11:27:50.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f169d3a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:27:50.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f169d3a8640 1 -- 172.21.15.135:0/734553213 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16981dc450 con 0x7f16980aeae0 2024-03-29T11:27:50.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.981+0000 7f169d3a8640 1 librados: init done 2024-03-29T11:27:50.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.983+0000 7f169d3a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:50.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.983+0000 7f169d3a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:50.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.983+0000 7f169d3a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:27:50.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.983+0000 7f169d3a8640 1 -- 172.21.15.135:0/734553213 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1698145cf0 con 0x7f16980aeae0 2024-03-29T11:27:50.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.983+0000 7f168f7fe640 1 -- 172.21.15.135:0/734553213 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1688001b10 con 0x7f16980aeae0 2024-03-29T11:27:50.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.983+0000 7f168f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:50.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.983+0000 7f168f7fe640 1 -- 172.21.15.135:0/734553213 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f16880038c0 con 0x7f16980aeae0 2024-03-29T11:27:50.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.983+0000 7f168f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:50.987 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.983+0000 7f168f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:50.987 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.983+0000 7f168f7fe640 10 monclient: dump: 2024-03-29T11:27:50.987 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:50.987 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:50.987 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:50.987 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:50.987 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:50.987 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:50.987 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:50.987 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:50.987 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:50.988 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:50.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.983+0000 7f1695ffb640 4 set_mon_vals no callback set 2024-03-29T11:27:50.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.985+0000 7f168f7fe640 1 -- 172.21.15.135:0/734553213 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f1688003a20 con 0x7f16980aeae0 2024-03-29T11:27:50.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.986+0000 7f168f7fe640 1 -- 172.21.15.135:0/734553213 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f16880849e0 con 0x7f16980aeae0 2024-03-29T11:27:50.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.986+0000 7f168f7fe640 10 client.4357.objecter ms_dispatch 0x7f1698000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:50.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.986+0000 7f168f7fe640 3 client.4357.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:50.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.986+0000 7f168f7fe640 3 client.4357.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:50.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.986+0000 7f168f7fe640 20 client.4357.objecter dump_active .. 0 homeless 2024-03-29T11:27:50.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.986+0000 7f168ffff640 10 monclient: get_auth_request con 0x7f167806e010 auth_method 0 2024-03-29T11:27:50.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.986+0000 7f168f7fe640 10 client.4357.objecter ms_handle_connect 0x7f167806e010 2024-03-29T11:27:50.989 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.992+0000 7f168f7fe640 1 -- 172.21.15.135:0/734553213 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f16880500f0 con 0x7f16980aeae0 2024-03-29T11:27:50.989 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.993+0000 7f168f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:50.989 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:50.993+0000 7f168f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:51.356 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:51.368+0000 7f169d3a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:51.356 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:51.368+0000 7f169d3a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:51.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:51.368+0000 7f169d3a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:27:51.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:51.368+0000 7f169d3a8640 1 -- 172.21.15.135:0/734553213 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1698144d80 con 0x7f16980aeae0 2024-03-29T11:27:51.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:51.369+0000 7f168f7fe640 1 -- 172.21.15.135:0/734553213 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f1698144d80 con 0x7f16980aeae0 2024-03-29T11:27:51.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:51.369+0000 7f168f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:51.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:51.369+0000 7f168f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:51.358 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:27:51.358 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:51.361 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:51.372+0000 7f169d3a8640 10 librados: watch_flush enter 2024-03-29T11:27:51.361 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:51.372+0000 7f169d3a8640 10 librados: watch_flush exit 2024-03-29T11:27:51.361 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:51.372+0000 7f169d3a8640 20 client.4357.objecter shutdown clearing up homeless session... 2024-03-29T11:27:51.361 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:51.372+0000 7f169d3a8640 10 client.4357.objecter successfully canceled tick 2024-03-29T11:27:51.361 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:51.372+0000 7f169d3a8640 1 -- 172.21.15.135:0/734553213 >> v1:172.21.15.135:6816/3469875285 conn(0x7f167806e010 legacy=0x7f16780704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:51.361 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:51.372+0000 7f169d3a8640 10 monclient: shutdown 2024-03-29T11:27:51.361 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:51.372+0000 7f169d3a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:51.362 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:51.372+0000 7f169d3a8640 1 -- 172.21.15.135:0/734553213 >> v1:172.21.15.194:6789/0 conn(0x7f16980aeae0 legacy=0x7f16981dbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:51.362 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:51.373+0000 7f169d3a8640 1 -- 172.21.15.135:0/734553213 shutdown_connections 2024-03-29T11:27:51.362 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:51.373+0000 7f169d3a8640 1 -- 172.21.15.135:0/734553213 >> 172.21.15.135:0/734553213 conn(0x7f1698066a00 msgr2=0x7f169813f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:51.362 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:51.373+0000 7f169d3a8640 1 -- 172.21.15.135:0/734553213 shutdown_connections 2024-03-29T11:27:51.362 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:51.373+0000 7f169d3a8640 1 -- 172.21.15.135:0/734553213 wait complete. 2024-03-29T11:27:51.362 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:51.373+0000 7f169d3a8640 1 librados: shutdown 2024-03-29T11:27:51.376 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:58.184 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:58.321 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.332+0000 7fd313e0d640 10 monclient: get_monmap_and_config 2024-03-29T11:27:58.321 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.332+0000 7fd313e0d640 10 monclient: build_initial_monmap 2024-03-29T11:27:58.321 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.332+0000 7fd313e0d640 10 monclient: monmap: 2024-03-29T11:27:58.321 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:27:58.321 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:58.322 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:27:58.333461+0000 2024-03-29T11:27:58.322 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:27:58.333461+0000 2024-03-29T11:27:58.322 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:58.322 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:58.322 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:27:58.322 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:27:58.322 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:27:58.322 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:58.329 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.333+0000 7fd313e0d640 1 Processor -- start 2024-03-29T11:27:58.329 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.334+0000 7fd313e0d640 1 -- start start 2024-03-29T11:27:58.329 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.334+0000 7fd313e0d640 10 monclient: init 2024-03-29T11:27:58.330 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.334+0000 7fd313e0d640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:58.330 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.334+0000 7fd313e0d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:27:58.330 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.334+0000 7fd313e0d640 10 monclient(hunting): picked mon.noname-b con 0x7fd30c1457f0 addr v1:172.21.15.135:6790/0 2024-03-29T11:27:58.330 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.334+0000 7fd313e0d640 10 monclient(hunting): picked mon.noname-c con 0x7fd30c13f650 addr v1:172.21.15.194:6789/0 2024-03-29T11:27:58.330 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.334+0000 7fd313e0d640 10 monclient(hunting): picked mon.noname-a con 0x7fd30c0a9d70 addr v1:172.21.15.135:6789/0 2024-03-29T11:27:58.330 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.334+0000 7fd313e0d640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd30c0aaef0 con 0x7fd30c0a9d70 2024-03-29T11:27:58.330 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.334+0000 7fd313e0d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd30c145bd0 con 0x7fd30c13f650 2024-03-29T11:27:58.330 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.334+0000 7fd313e0d640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd30c145eb0 con 0x7fd30c1457f0 2024-03-29T11:27:58.330 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.334+0000 7fd313e0d640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:58.331 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.334+0000 7fd313e0d640 10 monclient(hunting): authenticate will time out at 985.702974s 2024-03-29T11:27:58.331 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.335+0000 7fd3113a8640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fd30c13f650 0x7fd30c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:36330/0 (socket says 172.21.15.135:36330) 2024-03-29T11:27:58.331 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.335+0000 7fd3113a8640 1 -- 172.21.15.135:0/1091304728 learned_addr learned my addr 172.21.15.135:0/1091304728 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:27:58.331 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.335+0000 7fd310ba7640 1 -- 172.21.15.135:0/1091304728 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 525472040 0 0) 0x7fd30c145eb0 con 0x7fd30c1457f0 2024-03-29T11:27:58.331 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.335+0000 7fd310ba7640 10 monclient(hunting): _negotiate 2024-03-29T11:27:58.331 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.336+0000 7fd310ba7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:58.331 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.336+0000 7fd310ba7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:58.331 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.336+0000 7fd310ba7640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:27:58.331 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.336+0000 7fd310ba7640 1 -- 172.21.15.135:0/1091304728 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2f8003520 con 0x7fd30c1457f0 2024-03-29T11:27:58.331 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.336+0000 7fd310ba7640 1 -- 172.21.15.135:0/1091304728 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045184068 0 0) 0x7fd30c145bd0 con 0x7fd30c13f650 2024-03-29T11:27:58.331 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.336+0000 7fd310ba7640 10 monclient(hunting): _negotiate 2024-03-29T11:27:58.332 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.336+0000 7fd310ba7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:58.332 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.336+0000 7fd310ba7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:58.332 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.336+0000 7fd310ba7640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:27:58.332 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.336+0000 7fd310ba7640 1 -- 172.21.15.135:0/1091304728 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd30c145eb0 con 0x7fd30c13f650 2024-03-29T11:27:58.332 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.336+0000 7fd310ba7640 1 -- 172.21.15.135:0/1091304728 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3202540881 0 0) 0x7fd2f8003520 con 0x7fd30c1457f0 2024-03-29T11:27:58.332 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.336+0000 7fd310ba7640 1 -- 172.21.15.135:0/1091304728 >> v1:172.21.15.194:6789/0 conn(0x7fd30c13f650 legacy=0x7fd30c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:58.332 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.336+0000 7fd310ba7640 1 -- 172.21.15.135:0/1091304728 >> v1:172.21.15.135:6789/0 conn(0x7fd30c0a9d70 legacy=0x7fd30c14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:58.332 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.337+0000 7fd310ba7640 10 monclient: _finish_hunting 0 2024-03-29T11:27:58.332 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.337+0000 7fd310ba7640 1 monclient: found mon.noname-b 2024-03-29T11:27:58.332 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.337+0000 7fd310ba7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:58.332 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.337+0000 7fd310ba7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:27:58.333 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.337+0000 7fd310ba7640 1 -- 172.21.15.135:0/1091304728 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd30c146340 con 0x7fd30c1457f0 2024-03-29T11:27:58.333 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.337+0000 7fd310ba7640 10 monclient: _finish_auth 0 2024-03-29T11:27:58.333 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.337+0000 7fd310ba7640 10 monclient: _check_auth_tickets 2024-03-29T11:27:58.333 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.337+0000 7fd310ba7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:58.333 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.337+0000 7fd313e0d640 5 monclient: authenticate success, global_id 4349 2024-03-29T11:27:58.333 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.337+0000 7fd313e0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:58.333 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.337+0000 7fd310ba7640 1 -- 172.21.15.135:0/1091304728 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fd2fc001690 con 0x7fd30c1457f0 2024-03-29T11:27:58.333 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.337+0000 7fd310ba7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:58.333 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.337+0000 7fd310ba7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:58.333 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.337+0000 7fd310ba7640 10 monclient: dump: 2024-03-29T11:27:58.333 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:58.333 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:58.333 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:58.333 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:58.334 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:58.334 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:58.334 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:58.334 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:58.334 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:58.334 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:58.334 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.337+0000 7fd313e0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:58.334 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.337+0000 7fd310ba7640 1 -- 172.21.15.135:0/1091304728 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2fc0017f0 con 0x7fd30c1457f0 2024-03-29T11:27:58.334 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.337+0000 7fd310ba7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:58.334 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.337+0000 7fd310ba7640 1 -- 172.21.15.135:0/1091304728 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fd2fc001c00 con 0x7fd30c1457f0 2024-03-29T11:27:58.334 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.337+0000 7fd310ba7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:58.334 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.337+0000 7fd310ba7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:58.334 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.337+0000 7fd310ba7640 10 monclient: dump: 2024-03-29T11:27:58.334 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:58.335 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:58.335 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:58.335 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:58.335 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:58.335 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:58.335 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:58.335 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:58.335 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:58.335 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:58.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.337+0000 7fd313e0d640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:58.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.337+0000 7fd313e0d640 4 set_mon_vals no callback set 2024-03-29T11:27:58.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.339+0000 7fd313e0d640 10 monclient: shutdown 2024-03-29T11:27:58.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.339+0000 7fd313e0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:58.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.339+0000 7fd313e0d640 1 -- 172.21.15.135:0/1091304728 >> v1:172.21.15.135:6790/0 conn(0x7fd30c1457f0 legacy=0x7fd30c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:58.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.339+0000 7fd313e0d640 1 -- 172.21.15.135:0/1091304728 shutdown_connections 2024-03-29T11:27:58.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.339+0000 7fd313e0d640 1 -- 172.21.15.135:0/1091304728 >> 172.21.15.135:0/1091304728 conn(0x7fd30c066a00 msgr2=0x7fd30c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:58.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.340+0000 7fd313e0d640 1 -- 172.21.15.135:0/1091304728 shutdown_connections 2024-03-29T11:27:58.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.340+0000 7fd313e0d640 1 -- 172.21.15.135:0/1091304728 wait complete. 2024-03-29T11:27:58.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.342+0000 7fd313e0d640 10 monclient: build_initial_monmap 2024-03-29T11:27:58.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.342+0000 7fd313e0d640 10 monclient: monmap: 2024-03-29T11:27:58.342 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:27:58.342 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:58.342 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:27:58.342 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:27:58.342 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:58.342 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:58.342 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:27:58.342 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:27:58.343 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:27:58.343 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:58.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.342+0000 7fd313e0d640 1 librados: starting msgr at 2024-03-29T11:27:58.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.342+0000 7fd313e0d640 1 librados: starting objecter 2024-03-29T11:27:58.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.342+0000 7fd313e0d640 1 Processor -- start 2024-03-29T11:27:58.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.342+0000 7fd313e0d640 1 -- start start 2024-03-29T11:27:58.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.342+0000 7fd313e0d640 1 librados: setting wanted keys 2024-03-29T11:27:58.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.342+0000 7fd313e0d640 1 librados: calling monclient init 2024-03-29T11:27:58.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.342+0000 7fd313e0d640 10 monclient: init 2024-03-29T11:27:58.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.343+0000 7fd313e0d640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:58.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.343+0000 7fd313e0d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:58.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.343+0000 7fd313e0d640 10 monclient(hunting): picked mon.noname-b con 0x7fd30c0a9d70 addr v1:172.21.15.194:6789/0 2024-03-29T11:27:58.344 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.343+0000 7fd313e0d640 10 monclient(hunting): picked mon.noname-a con 0x7fd30c13f650 addr v1:172.21.15.135:6789/0 2024-03-29T11:27:58.344 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.343+0000 7fd313e0d640 10 monclient(hunting): picked mon.noname-c con 0x7fd30c1457f0 addr v1:172.21.15.135:6790/0 2024-03-29T11:27:58.344 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.343+0000 7fd313e0d640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd30c0ac780 con 0x7fd30c13f650 2024-03-29T11:27:58.344 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.343+0000 7fd313e0d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd30c137790 con 0x7fd30c0a9d70 2024-03-29T11:27:58.344 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.343+0000 7fd313e0d640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd30c0ab970 con 0x7fd30c1457f0 2024-03-29T11:27:58.344 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.343+0000 7fd313e0d640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:58.344 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.343+0000 7fd313e0d640 10 monclient(hunting): authenticate will time out at 985.711719s 2024-03-29T11:27:58.344 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.343+0000 7fd3123aa640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7fd30c1457f0 0x7fd30c137080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:52504/0 (socket says 172.21.15.135:52504) 2024-03-29T11:27:58.344 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.343+0000 7fd3123aa640 1 -- 172.21.15.135:0/1745808002 learned_addr learned my addr 172.21.15.135:0/1745808002 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:27:58.344 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.343+0000 7fd3027fc640 10 client.?.objecter ms_handle_connect 0x7fd30c1457f0 2024-03-29T11:27:58.344 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.343+0000 7fd3027fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:58.345 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.344+0000 7fd3027fc640 1 -- 172.21.15.135:0/1745808002 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615812850 0 0) 0x7fd30c0ab970 con 0x7fd30c1457f0 2024-03-29T11:27:58.345 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.344+0000 7fd3027fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:58.345 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.344+0000 7fd3027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:58.345 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.344+0000 7fd3027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:58.345 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.344+0000 7fd3027fc640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:27:58.346 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.344+0000 7fd3027fc640 1 -- 172.21.15.135:0/1745808002 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2f0004eb0 con 0x7fd30c1457f0 2024-03-29T11:27:58.346 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.344+0000 7fd3027fc640 1 -- 172.21.15.135:0/1745808002 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1507872983 0 0) 0x7fd2f0004eb0 con 0x7fd30c1457f0 2024-03-29T11:27:58.346 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.344+0000 7fd3027fc640 1 -- 172.21.15.135:0/1745808002 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd30c0ab970 con 0x7fd30c1457f0 2024-03-29T11:27:58.346 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.344+0000 7fd3027fc640 1 -- 172.21.15.135:0/1745808002 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fd304001c60 con 0x7fd30c1457f0 2024-03-29T11:27:58.346 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.344+0000 7fd3027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:58.346 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.344+0000 7fd3027fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:58.346 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.344+0000 7fd3027fc640 10 monclient(hunting): dump: 2024-03-29T11:27:58.347 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:58.347 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:58.347 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:58.347 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:58.347 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:58.347 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:58.347 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:58.347 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:58.347 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:58.347 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:58.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.344+0000 7fd3027fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:58.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.344+0000 7fd3027fc640 1 -- 172.21.15.135:0/1745808002 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 841068149 0 0) 0x7fd30c0ab970 con 0x7fd30c1457f0 2024-03-29T11:27:58.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.344+0000 7fd3027fc640 1 -- 172.21.15.135:0/1745808002 >> v1:172.21.15.194:6789/0 conn(0x7fd30c0a9d70 legacy=0x7fd30c140140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:58.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.345+0000 7fd3027fc640 1 -- 172.21.15.135:0/1745808002 >> v1:172.21.15.135:6789/0 conn(0x7fd30c13f650 legacy=0x7fd30c136970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:58.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.346+0000 7fd3027fc640 10 monclient: _finish_hunting 0 2024-03-29T11:27:58.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.346+0000 7fd3027fc640 1 monclient: found mon.c 2024-03-29T11:27:58.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.346+0000 7fd3027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:58.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.346+0000 7fd3027fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:27:58.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.346+0000 7fd3027fc640 1 -- 172.21.15.135:0/1745808002 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd30c0ac9f0 con 0x7fd30c1457f0 2024-03-29T11:27:58.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.346+0000 7fd3027fc640 10 monclient: _finish_auth 0 2024-03-29T11:27:58.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.346+0000 7fd3027fc640 10 monclient: _check_auth_tickets 2024-03-29T11:27:58.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.346+0000 7fd3027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:58.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.347+0000 7fd313e0d640 5 monclient: authenticate success, global_id 4352 2024-03-29T11:27:58.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.347+0000 7fd313e0d640 10 monclient: _renew_subs 2024-03-29T11:27:58.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.347+0000 7fd313e0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:27:58.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.347+0000 7fd313e0d640 1 -- 172.21.15.135:0/1745808002 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd30c0ab600 con 0x7fd30c1457f0 2024-03-29T11:27:58.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.347+0000 7fd313e0d640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:58.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.347+0000 7fd313e0d640 10 monclient: _renew_subs 2024-03-29T11:27:58.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.347+0000 7fd313e0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:27:58.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.347+0000 7fd313e0d640 1 -- 172.21.15.135:0/1745808002 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd30c137d10 con 0x7fd30c1457f0 2024-03-29T11:27:58.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.347+0000 7fd313e0d640 1 librados: init done 2024-03-29T11:27:58.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.347+0000 7fd3027fc640 1 -- 172.21.15.135:0/1745808002 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3040020c0 con 0x7fd30c1457f0 2024-03-29T11:27:58.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.347+0000 7fd3027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:58.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.347+0000 7fd3027fc640 1 -- 172.21.15.135:0/1745808002 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fd304003dc0 con 0x7fd30c1457f0 2024-03-29T11:27:58.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.347+0000 7fd3027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:58.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.347+0000 7fd3027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:58.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.347+0000 7fd3027fc640 10 monclient: dump: 2024-03-29T11:27:58.349 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:27:58.349 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:27:58.349 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:58.350 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:27:58.350 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:58.350 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:27:58.350 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:27:58.350 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:27:58.350 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:27:58.350 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:27:58.350 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.347+0000 7fd303fff640 4 set_mon_vals no callback set 2024-03-29T11:27:58.350 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.349+0000 7fd3027fc640 1 -- 172.21.15.135:0/1745808002 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fd304003f20 con 0x7fd30c1457f0 2024-03-29T11:27:58.350 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.349+0000 7fd313e0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:58.350 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.349+0000 7fd313e0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:58.350 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.349+0000 7fd313e0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:27:58.350 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.349+0000 7fd313e0d640 1 -- 172.21.15.135:0/1745808002 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f4005080 con 0x7fd30c1457f0 2024-03-29T11:27:58.350 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.350+0000 7fd3027fc640 1 -- 172.21.15.135:0/1745808002 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7fd304084e80 con 0x7fd30c1457f0 2024-03-29T11:27:58.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.350+0000 7fd3027fc640 10 client.4352.objecter ms_dispatch 0x7fd30c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:58.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.350+0000 7fd3027fc640 3 client.4352.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:58.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.350+0000 7fd3027fc640 3 client.4352.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:58.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.350+0000 7fd3027fc640 20 client.4352.objecter dump_active .. 0 homeless 2024-03-29T11:27:58.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.350+0000 7fd311ba9640 10 monclient: get_auth_request con 0x7fd2f006dbf0 auth_method 0 2024-03-29T11:27:58.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.350+0000 7fd3027fc640 10 client.4352.objecter ms_handle_connect 0x7fd2f006dbf0 2024-03-29T11:27:58.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.358+0000 7fd3027fc640 1 -- 172.21.15.135:0/1745808002 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd3040b28b0 con 0x7fd30c1457f0 2024-03-29T11:27:58.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.358+0000 7fd3027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:58.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.358+0000 7fd3027fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:58.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.663+0000 7fd313e0d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:58.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.663+0000 7fd313e0d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:58.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.663+0000 7fd313e0d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:27:58.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.663+0000 7fd313e0d640 1 -- 172.21.15.135:0/1745808002 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2f4005330 con 0x7fd30c1457f0 2024-03-29T11:27:58.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.663+0000 7fd3027fc640 1 -- 172.21.15.135:0/1745808002 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7fd304050760 con 0x7fd30c1457f0 2024-03-29T11:27:58.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.663+0000 7fd3027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:58.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.663+0000 7fd3027fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:58.652 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:27:58.653 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:58.655 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.666+0000 7fd313e0d640 10 librados: watch_flush enter 2024-03-29T11:27:58.655 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.666+0000 7fd313e0d640 10 librados: watch_flush exit 2024-03-29T11:27:58.655 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.666+0000 7fd313e0d640 20 client.4352.objecter shutdown clearing up homeless session... 2024-03-29T11:27:58.655 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.666+0000 7fd313e0d640 10 client.4352.objecter successfully canceled tick 2024-03-29T11:27:58.655 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.666+0000 7fd313e0d640 1 -- 172.21.15.135:0/1745808002 >> v1:172.21.15.135:6816/3469875285 conn(0x7fd2f006dbf0 legacy=0x7fd2f00700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:58.655 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.666+0000 7fd313e0d640 10 monclient: shutdown 2024-03-29T11:27:58.655 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.666+0000 7fd313e0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:58.655 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.666+0000 7fd313e0d640 1 -- 172.21.15.135:0/1745808002 >> v1:172.21.15.135:6790/0 conn(0x7fd30c1457f0 legacy=0x7fd30c137080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:58.655 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.666+0000 7fd313e0d640 1 -- 172.21.15.135:0/1745808002 shutdown_connections 2024-03-29T11:27:58.655 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.666+0000 7fd313e0d640 1 -- 172.21.15.135:0/1745808002 >> 172.21.15.135:0/1745808002 conn(0x7fd30c066a00 msgr2=0x7fd30c13aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:58.656 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.667+0000 7fd313e0d640 1 -- 172.21.15.135:0/1745808002 shutdown_connections 2024-03-29T11:27:58.656 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.667+0000 7fd313e0d640 1 -- 172.21.15.135:0/1745808002 wait complete. 2024-03-29T11:27:58.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:27:58.668+0000 7fd313e0d640 1 librados: shutdown 2024-03-29T11:27:58.671 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:05.479 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:05.630 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.640+0000 7f46bf539640 10 monclient: get_monmap_and_config 2024-03-29T11:28:05.630 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.640+0000 7f46bf539640 10 monclient: build_initial_monmap 2024-03-29T11:28:05.630 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.640+0000 7f46bf539640 10 monclient: monmap: 2024-03-29T11:28:05.630 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:28:05.630 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:05.630 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:28:05.642021+0000 2024-03-29T11:28:05.630 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:28:05.642021+0000 2024-03-29T11:28:05.630 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:05.631 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:05.631 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:28:05.631 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:28:05.631 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:28:05.631 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:05.632 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.641+0000 7f46bf539640 1 Processor -- start 2024-03-29T11:28:05.632 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.641+0000 7f46bf539640 1 -- start start 2024-03-29T11:28:05.632 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.641+0000 7f46bf539640 10 monclient: init 2024-03-29T11:28:05.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.642+0000 7f46bf539640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:05.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.642+0000 7f46bf539640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:05.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.642+0000 7f46bf539640 10 monclient(hunting): picked mon.noname-a con 0x7f46b80b0770 addr v1:172.21.15.135:6789/0 2024-03-29T11:28:05.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.642+0000 7f46bf539640 10 monclient(hunting): picked mon.noname-b con 0x7f46b814a700 addr v1:172.21.15.135:6790/0 2024-03-29T11:28:05.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.642+0000 7f46bf539640 10 monclient(hunting): picked mon.noname-c con 0x7f46b80ac5a0 addr v1:172.21.15.194:6789/0 2024-03-29T11:28:05.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.642+0000 7f46bf539640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b80aba70 con 0x7f46b80b0770 2024-03-29T11:28:05.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.642+0000 7f46bf539640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b80aecb0 con 0x7f46b80ac5a0 2024-03-29T11:28:05.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.642+0000 7f46bf539640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b80af150 con 0x7f46b814a700 2024-03-29T11:28:05.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.642+0000 7f46bf539640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:05.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.642+0000 7f46bf539640 10 monclient(hunting): authenticate will time out at 993.011766s 2024-03-29T11:28:05.635 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.645+0000 7f46bcad4640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f46b814a700 0x7f46b80abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:59688/0 (socket says 172.21.15.135:59688) 2024-03-29T11:28:05.635 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.645+0000 7f46bcad4640 1 -- 172.21.15.135:0/3708168110 learned_addr learned my addr 172.21.15.135:0/3708168110 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:28:05.635 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.645+0000 7f46b7fff640 1 -- 172.21.15.135:0/3708168110 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082155627 0 0) 0x7f46b80af150 con 0x7f46b814a700 2024-03-29T11:28:05.636 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.645+0000 7f46b7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:05.636 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.645+0000 7f46b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:05.636 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.645+0000 7f46b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:05.636 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.645+0000 7f46b7fff640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:28:05.636 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.646+0000 7f46b7fff640 1 -- 172.21.15.135:0/3708168110 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46a0003520 con 0x7f46b814a700 2024-03-29T11:28:05.639 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.646+0000 7f46b7fff640 1 -- 172.21.15.135:0/3708168110 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 821394496 0 0) 0x7f46a0003520 con 0x7f46b814a700 2024-03-29T11:28:05.639 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.646+0000 7f46b7fff640 1 -- 172.21.15.135:0/3708168110 >> v1:172.21.15.194:6789/0 conn(0x7f46b80ac5a0 legacy=0x7f46b8151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:05.639 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.646+0000 7f46b7fff640 1 -- 172.21.15.135:0/3708168110 >> v1:172.21.15.135:6789/0 conn(0x7f46b80b0770 legacy=0x7f46b8149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:05.639 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.646+0000 7f46b7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:05.639 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.646+0000 7f46b7fff640 1 monclient: found mon.noname-b 2024-03-29T11:28:05.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.646+0000 7f46b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:05.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.646+0000 7f46b7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:28:05.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.646+0000 7f46b7fff640 1 -- 172.21.15.135:0/3708168110 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b80ac9d0 con 0x7f46b814a700 2024-03-29T11:28:05.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.646+0000 7f46b7fff640 10 monclient: _finish_auth 0 2024-03-29T11:28:05.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.646+0000 7f46b7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:05.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.646+0000 7f46b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:05.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.647+0000 7f46bf539640 5 monclient: authenticate success, global_id 4355 2024-03-29T11:28:05.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.647+0000 7f46bf539640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:05.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.647+0000 7f46b7fff640 1 -- 172.21.15.135:0/3708168110 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f46b0001690 con 0x7f46b814a700 2024-03-29T11:28:05.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.647+0000 7f46b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:05.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.647+0000 7f46b7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:05.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.647+0000 7f46b7fff640 10 monclient: dump: 2024-03-29T11:28:05.640 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:05.640 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:05.640 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:05.641 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:05.641 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:05.641 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:05.641 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:05.641 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:05.641 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:05.641 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:05.641 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.647+0000 7f46bf539640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:05.641 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.647+0000 7f46b7fff640 1 -- 172.21.15.135:0/3708168110 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46b0001920 con 0x7f46b814a700 2024-03-29T11:28:05.641 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.647+0000 7f46b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:05.641 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.647+0000 7f46bf539640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:05.641 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.647+0000 7f46bf539640 4 set_mon_vals no callback set 2024-03-29T11:28:05.641 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.647+0000 7f46b7fff640 1 -- 172.21.15.135:0/3708168110 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f46b0001cf0 con 0x7f46b814a700 2024-03-29T11:28:05.641 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.647+0000 7f46b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:05.641 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.647+0000 7f46b7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:05.642 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.647+0000 7f46b7fff640 10 monclient: dump: 2024-03-29T11:28:05.642 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:05.642 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:05.642 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:05.642 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:05.642 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:05.642 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:05.642 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:05.642 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:05.642 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:05.642 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:05.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.648+0000 7f46bf539640 10 monclient: shutdown 2024-03-29T11:28:05.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.648+0000 7f46bf539640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:05.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.648+0000 7f46bf539640 1 -- 172.21.15.135:0/3708168110 >> v1:172.21.15.135:6790/0 conn(0x7f46b814a700 legacy=0x7f46b80abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:05.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.648+0000 7f46bf539640 1 -- 172.21.15.135:0/3708168110 shutdown_connections 2024-03-29T11:28:05.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.648+0000 7f46bf539640 1 -- 172.21.15.135:0/3708168110 >> 172.21.15.135:0/3708168110 conn(0x7f46b806b660 msgr2=0x7f46b806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:05.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.648+0000 7f46bf539640 1 -- 172.21.15.135:0/3708168110 shutdown_connections 2024-03-29T11:28:05.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.649+0000 7f46bf539640 1 -- 172.21.15.135:0/3708168110 wait complete. 2024-03-29T11:28:05.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.649+0000 7f46bf539640 10 monclient: build_initial_monmap 2024-03-29T11:28:05.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.649+0000 7f46bf539640 10 monclient: monmap: 2024-03-29T11:28:05.643 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:28:05.643 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:05.643 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:28:05.643 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:28:05.644 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:05.644 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:05.644 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:28:05.644 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:28:05.644 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:28:05.644 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:05.650 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.650+0000 7f46bf539640 1 librados: starting msgr at 2024-03-29T11:28:05.650 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.650+0000 7f46bf539640 1 librados: starting objecter 2024-03-29T11:28:05.650 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.650+0000 7f46bf539640 1 Processor -- start 2024-03-29T11:28:05.650 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.650+0000 7f46bf539640 1 -- start start 2024-03-29T11:28:05.650 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.650+0000 7f46bf539640 1 librados: setting wanted keys 2024-03-29T11:28:05.650 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.650+0000 7f46bf539640 1 librados: calling monclient init 2024-03-29T11:28:05.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.650+0000 7f46bf539640 10 monclient: init 2024-03-29T11:28:05.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.651+0000 7f46bf539640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:05.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.651+0000 7f46bf539640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:05.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.651+0000 7f46bf539640 10 monclient(hunting): picked mon.noname-b con 0x7f46b80ac5a0 addr v1:172.21.15.194:6789/0 2024-03-29T11:28:05.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.651+0000 7f46bf539640 10 monclient(hunting): picked mon.noname-a con 0x7f46b80b0770 addr v1:172.21.15.135:6789/0 2024-03-29T11:28:05.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.651+0000 7f46bf539640 10 monclient(hunting): picked mon.noname-c con 0x7f46b8145750 addr v1:172.21.15.135:6790/0 2024-03-29T11:28:05.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.651+0000 7f46bf539640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b81dea70 con 0x7f46b80b0770 2024-03-29T11:28:05.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.651+0000 7f46bf539640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b81281a0 con 0x7f46b80ac5a0 2024-03-29T11:28:05.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.651+0000 7f46bf539640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b8127210 con 0x7f46b8145750 2024-03-29T11:28:05.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.651+0000 7f46bf539640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:05.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.651+0000 7f46bf539640 10 monclient(hunting): authenticate will time out at 993.020525s 2024-03-29T11:28:05.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.651+0000 7f46bdad6640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f46b8145750 0x7f46b81def50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:60376/0 (socket says 172.21.15.135:60376) 2024-03-29T11:28:05.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.651+0000 7f46bdad6640 1 -- 172.21.15.135:0/2144663516 learned_addr learned my addr 172.21.15.135:0/2144663516 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:28:05.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.653+0000 7f46b5ffb640 10 client.?.objecter ms_handle_connect 0x7f46b8145750 2024-03-29T11:28:05.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.653+0000 7f46b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:05.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.654+0000 7f46b5ffb640 1 -- 172.21.15.135:0/2144663516 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2845914556 0 0) 0x7f46b8127210 con 0x7f46b8145750 2024-03-29T11:28:05.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.654+0000 7f46b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:05.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.654+0000 7f46b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:05.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.654+0000 7f46b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:05.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.654+0000 7f46b5ffb640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:28:05.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.654+0000 7f46b5ffb640 1 -- 172.21.15.135:0/2144663516 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46a4004eb0 con 0x7f46b8145750 2024-03-29T11:28:05.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.655+0000 7f46b5ffb640 10 client.?.objecter ms_handle_connect 0x7f46b80ac5a0 2024-03-29T11:28:05.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.655+0000 7f46b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:05.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.655+0000 7f46b5ffb640 10 client.?.objecter ms_handle_connect 0x7f46b80b0770 2024-03-29T11:28:05.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.655+0000 7f46b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:05.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.655+0000 7f46b5ffb640 1 -- 172.21.15.135:0/2144663516 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226361454 0 0) 0x7f46b81dea70 con 0x7f46b80b0770 2024-03-29T11:28:05.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:05.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:05.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:05.653 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 10 monclient(hunting): my global_id is 4365 2024-03-29T11:28:05.653 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 1 -- 172.21.15.135:0/2144663516 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46b8127210 con 0x7f46b80b0770 2024-03-29T11:28:05.653 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 1 -- 172.21.15.135:0/2144663516 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 550741619 0 0) 0x7f46a4004eb0 con 0x7f46b8145750 2024-03-29T11:28:05.653 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 1 -- 172.21.15.135:0/2144663516 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46b81dea70 con 0x7f46b8145750 2024-03-29T11:28:05.653 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 1 -- 172.21.15.135:0/2144663516 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f46a8001be0 con 0x7f46b8145750 2024-03-29T11:28:05.653 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:05.653 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:05.653 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 10 monclient(hunting): dump: 2024-03-29T11:28:05.653 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:05.653 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:05.653 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:05.653 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:05.653 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:05.653 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:05.653 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:05.654 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:05.654 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:05.654 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:05.654 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:05.654 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 1 -- 172.21.15.135:0/2144663516 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369668736 0 0) 0x7f46b81281a0 con 0x7f46b80ac5a0 2024-03-29T11:28:05.654 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:05.654 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:05.654 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:05.654 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:28:05.654 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 1 -- 172.21.15.135:0/2144663516 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46a4004eb0 con 0x7f46b80ac5a0 2024-03-29T11:28:05.654 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 1 -- 172.21.15.135:0/2144663516 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2802710455 0 0) 0x7f46a4004eb0 con 0x7f46b80ac5a0 2024-03-29T11:28:05.654 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 1 -- 172.21.15.135:0/2144663516 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46b81281a0 con 0x7f46b80ac5a0 2024-03-29T11:28:05.654 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 1 -- 172.21.15.135:0/2144663516 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f46ac001730 con 0x7f46b80ac5a0 2024-03-29T11:28:05.654 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:05.655 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:05.655 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 10 monclient(hunting): dump: 2024-03-29T11:28:05.655 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:05.655 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:05.655 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:05.655 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:05.655 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:05.655 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:05.655 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:05.655 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:05.655 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:05.655 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:05.655 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 1 -- 172.21.15.135:0/2144663516 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 310371727 0 0) 0x7f46b81281a0 con 0x7f46b80ac5a0 2024-03-29T11:28:05.655 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 1 -- 172.21.15.135:0/2144663516 >> v1:172.21.15.135:6790/0 conn(0x7f46b8145750 legacy=0x7f46b81def50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:05.655 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 1 -- 172.21.15.135:0/2144663516 >> v1:172.21.15.135:6789/0 conn(0x7f46b80b0770 legacy=0x7f46b8145040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:05.656 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:28:05.656 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 1 monclient: found mon.b 2024-03-29T11:28:05.656 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:05.656 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:28:05.656 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 1 -- 172.21.15.135:0/2144663516 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b81239d0 con 0x7f46b80ac5a0 2024-03-29T11:28:05.656 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:28:05.656 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:28:05.656 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:05.656 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46bf539640 5 monclient: authenticate success, global_id 4363 2024-03-29T11:28:05.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46bf539640 10 monclient: _renew_subs 2024-03-29T11:28:05.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46bf539640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:28:05.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.656+0000 7f46bf539640 1 -- 172.21.15.135:0/2144663516 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b81df660 con 0x7f46b80ac5a0 2024-03-29T11:28:05.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.657+0000 7f46bf539640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:05.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.657+0000 7f46bf539640 10 monclient: _renew_subs 2024-03-29T11:28:05.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.657+0000 7f46bf539640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:28:05.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.657+0000 7f46bf539640 1 -- 172.21.15.135:0/2144663516 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b81df9b0 con 0x7f46b80ac5a0 2024-03-29T11:28:05.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.657+0000 7f46bf539640 1 librados: init done 2024-03-29T11:28:05.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.658+0000 7f46b5ffb640 1 -- 172.21.15.135:0/2144663516 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46ac001c20 con 0x7f46b80ac5a0 2024-03-29T11:28:05.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.658+0000 7f46b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:05.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.658+0000 7f46b5ffb640 1 -- 172.21.15.135:0/2144663516 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f46ac003790 con 0x7f46b80ac5a0 2024-03-29T11:28:05.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.658+0000 7f46b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:05.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.658+0000 7f46b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:05.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.658+0000 7f46b5ffb640 10 monclient: dump: 2024-03-29T11:28:05.658 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:05.659 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:05.659 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:05.659 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:05.659 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:05.659 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:05.659 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:05.659 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:05.659 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:05.659 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:05.659 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.658+0000 7f46b77fe640 4 set_mon_vals no callback set 2024-03-29T11:28:05.659 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.658+0000 7f46bf539640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:05.659 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.659+0000 7f46bf539640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:05.659 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.659+0000 7f46bf539640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:28:05.659 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.659+0000 7f46bf539640 1 -- 172.21.15.135:0/2144663516 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4698005080 con 0x7f46b80ac5a0 2024-03-29T11:28:05.661 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.663+0000 7f46b5ffb640 1 -- 172.21.15.135:0/2144663516 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f46ac019760 con 0x7f46b80ac5a0 2024-03-29T11:28:05.661 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.665+0000 7f46bcad4640 10 monclient: get_auth_request con 0x7f46a406df80 auth_method 0 2024-03-29T11:28:05.661 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.666+0000 7f46b5ffb640 1 -- 172.21.15.135:0/2144663516 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f46ac083650 con 0x7f46b80ac5a0 2024-03-29T11:28:05.661 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.666+0000 7f46b5ffb640 10 client.4363.objecter ms_dispatch 0x7f46b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:05.661 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.666+0000 7f46b5ffb640 3 client.4363.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:05.662 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.666+0000 7f46b5ffb640 3 client.4363.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:05.662 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.666+0000 7f46b5ffb640 20 client.4363.objecter dump_active .. 0 homeless 2024-03-29T11:28:05.662 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.666+0000 7f46b5ffb640 10 client.4363.objecter ms_handle_connect 0x7f46a406df80 2024-03-29T11:28:05.662 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.669+0000 7f46b5ffb640 1 -- 172.21.15.135:0/2144663516 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f46ac04ef30 con 0x7f46b80ac5a0 2024-03-29T11:28:05.662 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.669+0000 7f46b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:05.662 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.669+0000 7f46b5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:05.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.988+0000 7f46bf539640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:05.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.988+0000 7f46bf539640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:05.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.988+0000 7f46bf539640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:28:05.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.988+0000 7f46bf539640 1 -- 172.21.15.135:0/2144663516 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4698005330 con 0x7f46b80ac5a0 2024-03-29T11:28:05.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.990+0000 7f46b5ffb640 1 -- 172.21.15.135:0/2144663516 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f46ac050e60 con 0x7f46b80ac5a0 2024-03-29T11:28:05.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.990+0000 7f46b5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:05.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.990+0000 7f46b5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:05.981 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:28:05.981 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:05.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.994+0000 7f46bf539640 10 librados: watch_flush enter 2024-03-29T11:28:05.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.995+0000 7f46bf539640 10 librados: watch_flush exit 2024-03-29T11:28:05.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.995+0000 7f46bf539640 20 client.4363.objecter shutdown clearing up homeless session... 2024-03-29T11:28:05.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.995+0000 7f46bf539640 10 client.4363.objecter successfully canceled tick 2024-03-29T11:28:05.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.995+0000 7f46bf539640 1 -- 172.21.15.135:0/2144663516 >> v1:172.21.15.135:6816/3469875285 conn(0x7f46a406df80 legacy=0x7f46a4070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:05.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.995+0000 7f46bf539640 10 monclient: shutdown 2024-03-29T11:28:05.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.995+0000 7f46bf539640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:05.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.995+0000 7f46bf539640 1 -- 172.21.15.135:0/2144663516 >> v1:172.21.15.194:6789/0 conn(0x7f46b80ac5a0 legacy=0x7f46b8126b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:05.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.995+0000 7f46bf539640 1 -- 172.21.15.135:0/2144663516 shutdown_connections 2024-03-29T11:28:05.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.995+0000 7f46bf539640 1 -- 172.21.15.135:0/2144663516 >> 172.21.15.135:0/2144663516 conn(0x7f46b806b660 msgr2=0x7f46b80abe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:05.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.997+0000 7f46bf539640 1 -- 172.21.15.135:0/2144663516 shutdown_connections 2024-03-29T11:28:05.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.997+0000 7f46bf539640 1 -- 172.21.15.135:0/2144663516 wait complete. 2024-03-29T11:28:05.987 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:05.997+0000 7f46bf539640 1 librados: shutdown 2024-03-29T11:28:06.005 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:12.812 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:12.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.974+0000 7f39b2024640 10 monclient: get_monmap_and_config 2024-03-29T11:28:12.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.974+0000 7f39b2024640 10 monclient: build_initial_monmap 2024-03-29T11:28:12.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.974+0000 7f39b2024640 10 monclient: monmap: 2024-03-29T11:28:12.965 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:28:12.965 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:12.965 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:28:12.976496+0000 2024-03-29T11:28:12.965 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:28:12.976496+0000 2024-03-29T11:28:12.965 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:12.965 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:12.965 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:28:12.965 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:28:12.966 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:28:12.966 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:12.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.975+0000 7f39b2024640 1 Processor -- start 2024-03-29T11:28:12.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.976+0000 7f39b2024640 1 -- start start 2024-03-29T11:28:12.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.976+0000 7f39b2024640 10 monclient: init 2024-03-29T11:28:12.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.976+0000 7f39b2024640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:12.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.976+0000 7f39b2024640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:12.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.976+0000 7f39b2024640 10 monclient(hunting): picked mon.noname-c con 0x7f39ac0aad20 addr v1:172.21.15.194:6789/0 2024-03-29T11:28:12.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.976+0000 7f39b2024640 10 monclient(hunting): picked mon.noname-b con 0x7f39ac06d8c0 addr v1:172.21.15.135:6790/0 2024-03-29T11:28:12.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.981+0000 7f39b1022640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f39ac0aad20 0x7f39ac0ac3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:46637/0 (socket says 172.21.15.135:46637) 2024-03-29T11:28:12.972 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.982+0000 7f39b0821640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f39ac06d8c0 0x7f39ac0a94e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:59493/0 (socket says 172.21.15.135:59493) 2024-03-29T11:28:12.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.982+0000 7f39b2024640 10 monclient(hunting): picked mon.noname-a con 0x7f39ac0afc40 addr v1:172.21.15.135:6789/0 2024-03-29T11:28:12.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.982+0000 7f39b2024640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39ac0af4d0 con 0x7f39ac0afc40 2024-03-29T11:28:12.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.982+0000 7f39b2024640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39ac0af7b0 con 0x7f39ac0aad20 2024-03-29T11:28:12.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.982+0000 7f39b2024640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39ac0acab0 con 0x7f39ac06d8c0 2024-03-29T11:28:12.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.982+0000 7f39b2024640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:12.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.982+0000 7f39b2024640 10 monclient(hunting): authenticate will time out at 1000.351930s 2024-03-29T11:28:12.974 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.984+0000 7f39b1823640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f39ac0afc40 0x7f39ac141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:54913/0 (socket says 172.21.15.135:54913) 2024-03-29T11:28:12.974 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.984+0000 7f39b1823640 1 -- 172.21.15.135:0/293512386 learned_addr learned my addr 172.21.15.135:0/293512386 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:28:12.975 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.985+0000 7f39a3fff640 1 -- 172.21.15.135:0/293512386 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 972902996 0 0) 0x7f39ac0af4d0 con 0x7f39ac0afc40 2024-03-29T11:28:12.975 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.985+0000 7f39a3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:12.975 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.985+0000 7f39a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:12.975 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.985+0000 7f39a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:12.975 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.985+0000 7f39a3fff640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:28:12.975 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.985+0000 7f39a3fff640 1 -- 172.21.15.135:0/293512386 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3998003520 con 0x7f39ac0afc40 2024-03-29T11:28:12.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.987+0000 7f39a3fff640 1 -- 172.21.15.135:0/293512386 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644628019 0 0) 0x7f39ac0acab0 con 0x7f39ac06d8c0 2024-03-29T11:28:12.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.987+0000 7f39a3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:12.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.987+0000 7f39a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:12.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.987+0000 7f39a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:12.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.987+0000 7f39a3fff640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:28:12.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.987+0000 7f39a3fff640 1 -- 172.21.15.135:0/293512386 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39ac0af4d0 con 0x7f39ac06d8c0 2024-03-29T11:28:12.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.987+0000 7f39a3fff640 1 -- 172.21.15.135:0/293512386 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886943859 0 0) 0x7f39ac0af7b0 con 0x7f39ac0aad20 2024-03-29T11:28:12.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.987+0000 7f39a3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:12.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.987+0000 7f39a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:12.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.987+0000 7f39a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:12.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.987+0000 7f39a3fff640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:28:12.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.987+0000 7f39a3fff640 1 -- 172.21.15.135:0/293512386 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39ac0acab0 con 0x7f39ac0aad20 2024-03-29T11:28:12.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.987+0000 7f39a3fff640 1 -- 172.21.15.135:0/293512386 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3434392358 0 0) 0x7f39ac0af4d0 con 0x7f39ac06d8c0 2024-03-29T11:28:12.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.987+0000 7f39a3fff640 1 -- 172.21.15.135:0/293512386 >> v1:172.21.15.194:6789/0 conn(0x7f39ac0aad20 legacy=0x7f39ac0ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:12.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.987+0000 7f39a3fff640 1 -- 172.21.15.135:0/293512386 >> v1:172.21.15.135:6789/0 conn(0x7f39ac0afc40 legacy=0x7f39ac141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:12.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.987+0000 7f39a3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:12.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.987+0000 7f39a3fff640 1 monclient: found mon.noname-b 2024-03-29T11:28:12.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.987+0000 7f39a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:12.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.987+0000 7f39a3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:28:12.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.987+0000 7f39a3fff640 1 -- 172.21.15.135:0/293512386 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ac0aba30 con 0x7f39ac06d8c0 2024-03-29T11:28:12.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.987+0000 7f39a3fff640 10 monclient: _finish_auth 0 2024-03-29T11:28:12.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.987+0000 7f39a3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:12.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.987+0000 7f39a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:12.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.987+0000 7f39a3fff640 1 -- 172.21.15.135:0/293512386 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f399c0015a0 con 0x7f39ac06d8c0 2024-03-29T11:28:12.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.987+0000 7f39a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:12.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.988+0000 7f39a3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:12.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.988+0000 7f39a3fff640 10 monclient: dump: 2024-03-29T11:28:12.980 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:12.980 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:12.980 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:12.980 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:12.980 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:12.980 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:12.981 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:12.981 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:12.981 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:12.981 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:12.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.988+0000 7f39b2024640 5 monclient: authenticate success, global_id 4361 2024-03-29T11:28:12.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.988+0000 7f39b2024640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:12.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.988+0000 7f39a3fff640 1 -- 172.21.15.135:0/293512386 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f399c001700 con 0x7f39ac06d8c0 2024-03-29T11:28:12.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.988+0000 7f39a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:12.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.988+0000 7f39a3fff640 1 -- 172.21.15.135:0/293512386 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f399c001a90 con 0x7f39ac06d8c0 2024-03-29T11:28:12.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.988+0000 7f39a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:12.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.988+0000 7f39a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:12.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.988+0000 7f39a3fff640 10 monclient: dump: 2024-03-29T11:28:12.981 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:12.981 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:12.982 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:12.982 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:12.982 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:12.982 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:12.982 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:12.982 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:12.982 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:12.982 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:12.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.988+0000 7f39b2024640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:12.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.988+0000 7f39b2024640 4 set_mon_vals no callback set 2024-03-29T11:28:12.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.989+0000 7f39b2024640 10 monclient: shutdown 2024-03-29T11:28:12.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.989+0000 7f39b2024640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:12.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.989+0000 7f39b2024640 1 -- 172.21.15.135:0/293512386 >> v1:172.21.15.135:6790/0 conn(0x7f39ac06d8c0 legacy=0x7f39ac0a94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:12.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.992+0000 7f39b2024640 1 -- 172.21.15.135:0/293512386 shutdown_connections 2024-03-29T11:28:12.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.992+0000 7f39b2024640 1 -- 172.21.15.135:0/293512386 >> 172.21.15.135:0/293512386 conn(0x7f39ac0673e0 msgr2=0x7f39ac0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:12.984 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.992+0000 7f39b2024640 1 -- 172.21.15.135:0/293512386 shutdown_connections 2024-03-29T11:28:12.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.993+0000 7f39b2024640 1 -- 172.21.15.135:0/293512386 wait complete. 2024-03-29T11:28:12.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.994+0000 7f39b2024640 10 monclient: build_initial_monmap 2024-03-29T11:28:12.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.994+0000 7f39b2024640 10 monclient: monmap: 2024-03-29T11:28:12.985 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:28:12.985 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:12.985 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:28:12.985 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:28:12.986 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:12.986 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:12.986 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:28:12.986 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:28:12.986 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:28:12.986 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:12.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.995+0000 7f39b2024640 1 librados: starting msgr at 2024-03-29T11:28:12.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.995+0000 7f39b2024640 1 librados: starting objecter 2024-03-29T11:28:12.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.995+0000 7f39b2024640 1 Processor -- start 2024-03-29T11:28:12.987 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.995+0000 7f39b2024640 1 -- start start 2024-03-29T11:28:12.987 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.995+0000 7f39b2024640 1 librados: setting wanted keys 2024-03-29T11:28:12.987 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.995+0000 7f39b2024640 1 librados: calling monclient init 2024-03-29T11:28:12.987 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.995+0000 7f39b2024640 10 monclient: init 2024-03-29T11:28:12.987 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.996+0000 7f39b2024640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:12.987 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:12.996+0000 7f39b2024640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:12.989 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.000+0000 7f39b2024640 10 monclient(hunting): picked mon.noname-c con 0x7f39ac06d8c0 addr v1:172.21.15.135:6790/0 2024-03-29T11:28:12.991 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.001+0000 7f39b2024640 10 monclient(hunting): picked mon.noname-a con 0x7f39ac0aad20 addr v1:172.21.15.135:6789/0 2024-03-29T11:28:12.991 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.001+0000 7f39b2024640 10 monclient(hunting): picked mon.noname-b con 0x7f39ac0afc40 addr v1:172.21.15.194:6789/0 2024-03-29T11:28:12.991 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.001+0000 7f39b2024640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39ac06f480 con 0x7f39ac0aad20 2024-03-29T11:28:12.992 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.001+0000 7f39b2024640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39ac06f760 con 0x7f39ac0afc40 2024-03-29T11:28:12.992 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.001+0000 7f39b2024640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39ac06fa40 con 0x7f39ac06d8c0 2024-03-29T11:28:12.992 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.001+0000 7f39b2024640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:12.992 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.001+0000 7f39b2024640 10 monclient(hunting): authenticate will time out at 1000.370384s 2024-03-29T11:28:12.992 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.001+0000 7f39b1022640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f39ac06d8c0 0x7f39ac070140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:59625/0 (socket says 172.21.15.135:59625) 2024-03-29T11:28:12.992 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.001+0000 7f39b1022640 1 -- 172.21.15.135:0/4230413475 learned_addr learned my addr 172.21.15.135:0/4230413475 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:28:12.992 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.002+0000 7f39a1ffb640 10 client.?.objecter ms_handle_connect 0x7f39ac06d8c0 2024-03-29T11:28:12.992 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.002+0000 7f39a1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:12.993 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.003+0000 7f39a1ffb640 1 -- 172.21.15.135:0/4230413475 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3371135749 0 0) 0x7f39ac06fa40 con 0x7f39ac06d8c0 2024-03-29T11:28:12.993 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.003+0000 7f39a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:12.993 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.003+0000 7f39a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:12.993 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.003+0000 7f39a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:12.993 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.003+0000 7f39a1ffb640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:28:12.993 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.003+0000 7f39a1ffb640 1 -- 172.21.15.135:0/4230413475 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3990004eb0 con 0x7f39ac06d8c0 2024-03-29T11:28:12.993 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.003+0000 7f39a1ffb640 1 -- 172.21.15.135:0/4230413475 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3840347871 0 0) 0x7f3990004eb0 con 0x7f39ac06d8c0 2024-03-29T11:28:12.993 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.003+0000 7f39a1ffb640 1 -- 172.21.15.135:0/4230413475 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39ac06fa40 con 0x7f39ac06d8c0 2024-03-29T11:28:12.993 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.003+0000 7f39a1ffb640 1 -- 172.21.15.135:0/4230413475 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f39a80018f0 con 0x7f39ac06d8c0 2024-03-29T11:28:12.994 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.003+0000 7f39a1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:12.994 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.003+0000 7f39a1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:12.994 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.003+0000 7f39a1ffb640 10 monclient(hunting): dump: 2024-03-29T11:28:12.994 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:12.994 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:12.994 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:12.994 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:12.994 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:12.994 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:12.994 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:12.994 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:12.994 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:12.994 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:12.995 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.003+0000 7f39a1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:12.995 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.004+0000 7f39a1ffb640 1 -- 172.21.15.135:0/4230413475 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1432134505 0 0) 0x7f39ac06fa40 con 0x7f39ac06d8c0 2024-03-29T11:28:12.995 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.004+0000 7f39a1ffb640 1 -- 172.21.15.135:0/4230413475 >> v1:172.21.15.194:6789/0 conn(0x7f39ac0afc40 legacy=0x7f39ac139c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:12.998 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.007+0000 7f39a1ffb640 1 -- 172.21.15.135:0/4230413475 >> v1:172.21.15.135:6789/0 conn(0x7f39ac0aad20 legacy=0x7f39ac139540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:12.998 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.007+0000 7f39a1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:28:12.998 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.007+0000 7f39a1ffb640 1 monclient: found mon.c 2024-03-29T11:28:12.998 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.007+0000 7f39a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:12.998 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.007+0000 7f39a1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:28:12.999 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.007+0000 7f39a1ffb640 1 -- 172.21.15.135:0/4230413475 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ac06dd10 con 0x7f39ac06d8c0 2024-03-29T11:28:12.999 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.007+0000 7f39a1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:28:12.999 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.007+0000 7f39a1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:28:12.999 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.007+0000 7f39a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:12.999 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.007+0000 7f39a1ffb640 10 client.?.objecter ms_handle_connect 0x7f39ac0aad20 2024-03-29T11:28:12.999 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.007+0000 7f39b2024640 5 monclient: authenticate success, global_id 4364 2024-03-29T11:28:12.999 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.007+0000 7f39b2024640 10 monclient: _renew_subs 2024-03-29T11:28:12.999 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.007+0000 7f39b2024640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:28:12.999 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.008+0000 7f39b2024640 1 -- 172.21.15.135:0/4230413475 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39ac06ebc0 con 0x7f39ac06d8c0 2024-03-29T11:28:12.999 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.008+0000 7f39b2024640 10 client.4364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:12.999 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.008+0000 7f39b2024640 10 monclient: _renew_subs 2024-03-29T11:28:12.999 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.008+0000 7f39b2024640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:28:12.999 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.008+0000 7f39b2024640 1 -- 172.21.15.135:0/4230413475 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39ac1c9b00 con 0x7f39ac06d8c0 2024-03-29T11:28:12.999 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.008+0000 7f39b2024640 1 librados: init done 2024-03-29T11:28:13.000 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.008+0000 7f39a1ffb640 10 client.4364.objecter resend_mon_ops 2024-03-29T11:28:13.000 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.011+0000 7f39b2024640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:13.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.011+0000 7f39b2024640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:13.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.011+0000 7f39b2024640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:28:13.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.011+0000 7f39b2024640 1 -- 172.21.15.135:0/4230413475 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39ac1c9e50 con 0x7f39ac06d8c0 2024-03-29T11:28:13.003 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.012+0000 7f39a1ffb640 1 -- 172.21.15.135:0/4230413475 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39a8001d60 con 0x7f39ac06d8c0 2024-03-29T11:28:13.003 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.012+0000 7f39a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:13.003 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.012+0000 7f39a1ffb640 1 -- 172.21.15.135:0/4230413475 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f39a8004c30 con 0x7f39ac06d8c0 2024-03-29T11:28:13.003 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.012+0000 7f39a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:13.003 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.012+0000 7f39a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:13.003 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.012+0000 7f39a1ffb640 10 monclient: dump: 2024-03-29T11:28:13.003 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:13.004 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:13.004 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:13.004 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:13.004 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:13.004 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:13.004 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:13.004 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:13.004 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:13.004 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:13.004 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.013+0000 7f39a1ffb640 1 -- 172.21.15.135:0/4230413475 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f39a8004e50 con 0x7f39ac06d8c0 2024-03-29T11:28:13.005 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.014+0000 7f39a37fe640 4 set_mon_vals no callback set 2024-03-29T11:28:13.005 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.014+0000 7f39b0821640 10 monclient: get_auth_request con 0x7f399006dbd0 auth_method 0 2024-03-29T11:28:13.005 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.015+0000 7f39a1ffb640 10 client.4364.objecter ms_handle_connect 0x7f399006dbd0 2024-03-29T11:28:13.005 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.015+0000 7f39a1ffb640 1 -- 172.21.15.135:0/4230413475 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f39a8084b30 con 0x7f39ac06d8c0 2024-03-29T11:28:13.005 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.015+0000 7f39a1ffb640 10 client.4364.objecter ms_dispatch 0x7f39ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:13.006 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.015+0000 7f39a1ffb640 3 client.4364.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:13.006 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.015+0000 7f39a1ffb640 3 client.4364.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:13.006 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.015+0000 7f39a1ffb640 20 client.4364.objecter dump_active .. 0 homeless 2024-03-29T11:28:13.013 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.023+0000 7f39a1ffb640 1 -- 172.21.15.135:0/4230413475 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f39a8050410 con 0x7f39ac06d8c0 2024-03-29T11:28:13.013 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.023+0000 7f39a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:13.013 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.023+0000 7f39a1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:13.329 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.339+0000 7f39b2024640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:13.329 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.339+0000 7f39b2024640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:13.329 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.339+0000 7f39b2024640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:28:13.329 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.339+0000 7f39b2024640 1 -- 172.21.15.135:0/4230413475 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39ac04f730 con 0x7f39ac06d8c0 2024-03-29T11:28:13.331 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.341+0000 7f39a1ffb640 1 -- 172.21.15.135:0/4230413475 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f39a8052340 con 0x7f39ac06d8c0 2024-03-29T11:28:13.331 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.341+0000 7f39a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:13.331 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.341+0000 7f39a1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:13.332 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:28:13.332 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:13.334 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.345+0000 7f39b2024640 10 librados: watch_flush enter 2024-03-29T11:28:13.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.345+0000 7f39b2024640 10 librados: watch_flush exit 2024-03-29T11:28:13.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.345+0000 7f39b2024640 20 client.4364.objecter shutdown clearing up homeless session... 2024-03-29T11:28:13.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.345+0000 7f39b2024640 10 client.4364.objecter successfully canceled tick 2024-03-29T11:28:13.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.345+0000 7f39b2024640 1 -- 172.21.15.135:0/4230413475 >> v1:172.21.15.135:6816/3469875285 conn(0x7f399006dbd0 legacy=0x7f3990070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:13.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.345+0000 7f39b2024640 10 monclient: shutdown 2024-03-29T11:28:13.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.345+0000 7f39b2024640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:13.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.345+0000 7f39b2024640 1 -- 172.21.15.135:0/4230413475 >> v1:172.21.15.135:6790/0 conn(0x7f39ac06d8c0 legacy=0x7f39ac070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:13.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.345+0000 7f39b2024640 1 -- 172.21.15.135:0/4230413475 shutdown_connections 2024-03-29T11:28:13.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.345+0000 7f39b2024640 1 -- 172.21.15.135:0/4230413475 >> 172.21.15.135:0/4230413475 conn(0x7f39ac0673e0 msgr2=0x7f39ac1417f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:13.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.346+0000 7f39b2024640 1 -- 172.21.15.135:0/4230413475 shutdown_connections 2024-03-29T11:28:13.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.346+0000 7f39b2024640 1 -- 172.21.15.135:0/4230413475 wait complete. 2024-03-29T11:28:13.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:13.347+0000 7f39b2024640 1 librados: shutdown 2024-03-29T11:28:13.356 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:20.164 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:20.318 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.327+0000 7fa1729a0640 10 monclient: get_monmap_and_config 2024-03-29T11:28:20.318 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.327+0000 7fa1729a0640 10 monclient: build_initial_monmap 2024-03-29T11:28:20.318 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.327+0000 7fa1729a0640 10 monclient: monmap: 2024-03-29T11:28:20.318 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:28:20.318 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:20.318 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:28:20.329541+0000 2024-03-29T11:28:20.319 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:28:20.329541+0000 2024-03-29T11:28:20.319 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:20.319 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:20.319 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:28:20.319 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:28:20.319 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:28:20.319 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:20.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.328+0000 7fa1729a0640 1 Processor -- start 2024-03-29T11:28:20.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.329+0000 7fa1729a0640 1 -- start start 2024-03-29T11:28:20.321 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.329+0000 7fa1729a0640 10 monclient: init 2024-03-29T11:28:20.321 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.330+0000 7fa1729a0640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:20.321 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.330+0000 7fa1729a0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:20.321 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.330+0000 7fa1729a0640 10 monclient(hunting): picked mon.noname-c con 0x7fa164006e80 addr v1:172.21.15.194:6789/0 2024-03-29T11:28:20.321 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.330+0000 7fa1729a0640 10 monclient(hunting): picked mon.noname-a con 0x7fa1640c9630 addr v1:172.21.15.135:6789/0 2024-03-29T11:28:20.321 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.330+0000 7fa1729a0640 10 monclient(hunting): picked mon.noname-b con 0x7fa164008580 addr v1:172.21.15.135:6790/0 2024-03-29T11:28:20.321 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.330+0000 7fa1729a0640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa16402be90 con 0x7fa1640c9630 2024-03-29T11:28:20.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.330+0000 7fa1729a0640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa16402c170 con 0x7fa164006e80 2024-03-29T11:28:20.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.330+0000 7fa1729a0640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa16402c450 con 0x7fa164008580 2024-03-29T11:28:20.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.330+0000 7fa1729a0640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:20.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.330+0000 7fa1729a0640 10 monclient(hunting): authenticate will time out at 1007.699377s 2024-03-29T11:28:20.323 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.333+0000 7fa17199e640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fa164006e80 0x7fa1640c8f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:35353/0 (socket says 172.21.15.135:35353) 2024-03-29T11:28:20.324 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.333+0000 7fa17199e640 1 -- 172.21.15.135:0/90768869 learned_addr learned my addr 172.21.15.135:0/90768869 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:28:20.328 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.335+0000 7fa17099c640 1 -- 172.21.15.135:0/90768869 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2279213296 0 0) 0x7fa16402c170 con 0x7fa164006e80 2024-03-29T11:28:20.328 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.336+0000 7fa17099c640 10 monclient(hunting): _negotiate 2024-03-29T11:28:20.328 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.336+0000 7fa17099c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:20.328 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.336+0000 7fa17099c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:20.329 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.336+0000 7fa17099c640 10 monclient(hunting): my global_id is 4369 2024-03-29T11:28:20.329 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.336+0000 7fa17099c640 1 -- 172.21.15.135:0/90768869 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa158003520 con 0x7fa164006e80 2024-03-29T11:28:20.329 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.337+0000 7fa17099c640 1 -- 172.21.15.135:0/90768869 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 261061181 0 0) 0x7fa158003520 con 0x7fa164006e80 2024-03-29T11:28:20.329 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.337+0000 7fa17099c640 1 -- 172.21.15.135:0/90768869 >> v1:172.21.15.135:6790/0 conn(0x7fa164008580 legacy=0x7fa1640d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:20.330 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.337+0000 7fa17099c640 1 -- 172.21.15.135:0/90768869 >> v1:172.21.15.135:6789/0 conn(0x7fa1640c9630 legacy=0x7fa1640cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.330 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.337+0000 7fa17099c640 10 monclient: _finish_hunting 0 2024-03-29T11:28:20.330 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.337+0000 7fa17099c640 1 monclient: found mon.noname-c 2024-03-29T11:28:20.330 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.337+0000 7fa17099c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:20.331 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.337+0000 7fa17099c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:28:20.331 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.337+0000 7fa17099c640 1 -- 172.21.15.135:0/90768869 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa164008a60 con 0x7fa164006e80 2024-03-29T11:28:20.331 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.337+0000 7fa17099c640 10 monclient: _finish_auth 0 2024-03-29T11:28:20.331 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.337+0000 7fa17099c640 10 monclient: _check_auth_tickets 2024-03-29T11:28:20.331 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.337+0000 7fa17099c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:20.331 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.337+0000 7fa17099c640 1 -- 172.21.15.135:0/90768869 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fa15c001690 con 0x7fa164006e80 2024-03-29T11:28:20.331 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.337+0000 7fa17099c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:20.332 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.337+0000 7fa17099c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:20.332 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.337+0000 7fa17099c640 10 monclient: dump: 2024-03-29T11:28:20.332 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:20.332 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:20.332 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:20.332 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:20.333 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:20.333 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:20.333 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:20.333 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:20.333 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:20.333 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:20.333 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.337+0000 7fa1729a0640 5 monclient: authenticate success, global_id 4369 2024-03-29T11:28:20.334 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.337+0000 7fa1729a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:20.334 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.337+0000 7fa17099c640 1 -- 172.21.15.135:0/90768869 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa15c0017f0 con 0x7fa164006e80 2024-03-29T11:28:20.334 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.337+0000 7fa17099c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:20.334 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.337+0000 7fa17099c640 1 -- 172.21.15.135:0/90768869 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fa15c001c40 con 0x7fa164006e80 2024-03-29T11:28:20.334 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.337+0000 7fa17099c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:20.334 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.337+0000 7fa17099c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:20.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.337+0000 7fa17099c640 10 monclient: dump: 2024-03-29T11:28:20.335 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:20.335 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:20.335 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:20.335 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:20.335 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:20.336 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:20.336 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:20.336 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:20.336 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:20.336 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:20.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.337+0000 7fa1729a0640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:20.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.337+0000 7fa1729a0640 4 set_mon_vals no callback set 2024-03-29T11:28:20.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.338+0000 7fa1729a0640 10 monclient: shutdown 2024-03-29T11:28:20.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.338+0000 7fa1729a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:20.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.338+0000 7fa1729a0640 1 -- 172.21.15.135:0/90768869 >> v1:172.21.15.194:6789/0 conn(0x7fa164006e80 legacy=0x7fa1640c8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.339+0000 7fa1729a0640 1 -- 172.21.15.135:0/90768869 shutdown_connections 2024-03-29T11:28:20.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.339+0000 7fa1729a0640 1 -- 172.21.15.135:0/90768869 >> 172.21.15.135:0/90768869 conn(0x7fa164005d00 msgr2=0x7fa1640060e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:20.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.340+0000 7fa1729a0640 1 -- 172.21.15.135:0/90768869 shutdown_connections 2024-03-29T11:28:20.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.340+0000 7fa1729a0640 1 -- 172.21.15.135:0/90768869 wait complete. 2024-03-29T11:28:20.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.341+0000 7fa1729a0640 10 monclient: build_initial_monmap 2024-03-29T11:28:20.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.341+0000 7fa1729a0640 10 monclient: monmap: 2024-03-29T11:28:20.340 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:28:20.340 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:20.340 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:28:20.340 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:28:20.340 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:20.341 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:20.341 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:28:20.341 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:28:20.341 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:28:20.341 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:20.341 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.342+0000 7fa1729a0640 1 librados: starting msgr at 2024-03-29T11:28:20.341 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.342+0000 7fa1729a0640 1 librados: starting objecter 2024-03-29T11:28:20.341 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.342+0000 7fa1729a0640 1 Processor -- start 2024-03-29T11:28:20.341 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.342+0000 7fa1729a0640 1 -- start start 2024-03-29T11:28:20.341 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.342+0000 7fa1729a0640 1 librados: setting wanted keys 2024-03-29T11:28:20.341 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.342+0000 7fa1729a0640 1 librados: calling monclient init 2024-03-29T11:28:20.341 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.342+0000 7fa1729a0640 10 monclient: init 2024-03-29T11:28:20.341 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.347+0000 7fa1729a0640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:20.341 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.347+0000 7fa1729a0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:20.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.347+0000 7fa1729a0640 10 monclient(hunting): picked mon.noname-a con 0x7fa164006e80 addr v1:172.21.15.135:6789/0 2024-03-29T11:28:20.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.347+0000 7fa1729a0640 10 monclient(hunting): picked mon.noname-b con 0x7fa164008580 addr v1:172.21.15.194:6789/0 2024-03-29T11:28:20.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.347+0000 7fa1729a0640 10 monclient(hunting): picked mon.noname-c con 0x7fa1640c9630 addr v1:172.21.15.135:6790/0 2024-03-29T11:28:20.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.347+0000 7fa1729a0640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1640bdea0 con 0x7fa164006e80 2024-03-29T11:28:20.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.347+0000 7fa1729a0640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1640cee90 con 0x7fa164008580 2024-03-29T11:28:20.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.347+0000 7fa1729a0640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1640ceab0 con 0x7fa1640c9630 2024-03-29T11:28:20.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.347+0000 7fa1729a0640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:20.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.347+0000 7fa1729a0640 10 monclient(hunting): authenticate will time out at 1007.716495s 2024-03-29T11:28:20.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.351+0000 7fa17199e640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fa164006e80 0x7fa1640cd1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:55921/0 (socket says 172.21.15.135:55921) 2024-03-29T11:28:20.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.351+0000 7fa17199e640 1 -- 172.21.15.135:0/1105143655 learned_addr learned my addr 172.21.15.135:0/1105143655 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:28:20.346 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.354+0000 7fa1627fc640 10 client.?.objecter ms_handle_connect 0x7fa164008580 2024-03-29T11:28:20.346 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.354+0000 7fa1627fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:20.346 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.354+0000 7fa1627fc640 10 client.?.objecter ms_handle_connect 0x7fa164006e80 2024-03-29T11:28:20.346 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.354+0000 7fa1627fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:20.346 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.354+0000 7fa1627fc640 1 -- 172.21.15.135:0/1105143655 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022381770 0 0) 0x7fa1640bdea0 con 0x7fa164006e80 2024-03-29T11:28:20.346 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.354+0000 7fa1627fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:20.346 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.354+0000 7fa1627fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:20.346 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.354+0000 7fa1627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:20.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.355+0000 7fa1627fc640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:28:20.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.355+0000 7fa1627fc640 1 -- 172.21.15.135:0/1105143655 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa150004eb0 con 0x7fa164006e80 2024-03-29T11:28:20.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.355+0000 7fa1627fc640 1 -- 172.21.15.135:0/1105143655 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4107691553 0 0) 0x7fa1640cee90 con 0x7fa164008580 2024-03-29T11:28:20.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.355+0000 7fa1627fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:20.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.355+0000 7fa1627fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:20.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.355+0000 7fa1627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:20.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.355+0000 7fa1627fc640 10 monclient(hunting): my global_id is 4372 2024-03-29T11:28:20.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.355+0000 7fa1627fc640 1 -- 172.21.15.135:0/1105143655 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1640bdea0 con 0x7fa164008580 2024-03-29T11:28:20.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.355+0000 7fa1627fc640 10 client.?.objecter ms_handle_connect 0x7fa1640c9630 2024-03-29T11:28:20.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.355+0000 7fa1627fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:20.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1627fc640 1 -- 172.21.15.135:0/1105143655 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1488861492 0 0) 0x7fa1640bdea0 con 0x7fa164008580 2024-03-29T11:28:20.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1627fc640 1 -- 172.21.15.135:0/1105143655 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1640cee90 con 0x7fa164008580 2024-03-29T11:28:20.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1627fc640 1 -- 172.21.15.135:0/1105143655 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3504173116 0 0) 0x7fa150004eb0 con 0x7fa164006e80 2024-03-29T11:28:20.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1627fc640 1 -- 172.21.15.135:0/1105143655 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1640bdea0 con 0x7fa164006e80 2024-03-29T11:28:20.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1627fc640 1 -- 172.21.15.135:0/1105143655 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fa168001890 con 0x7fa164008580 2024-03-29T11:28:20.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:20.350 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1627fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:20.350 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1627fc640 10 monclient(hunting): dump: 2024-03-29T11:28:20.350 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:20.350 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:20.350 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:20.350 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:20.350 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:20.350 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:20.350 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:20.350 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:20.350 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:20.350 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:20.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1627fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:20.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1627fc640 1 -- 172.21.15.135:0/1105143655 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fa15c001b00 con 0x7fa164006e80 2024-03-29T11:28:20.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:20.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1627fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:20.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1627fc640 10 monclient(hunting): dump: 2024-03-29T11:28:20.351 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:20.351 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:20.351 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:20.351 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:20.351 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:20.351 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:20.352 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:20.352 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:20.352 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:20.352 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:20.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1627fc640 1 -- 172.21.15.135:0/1105143655 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3038705811 0 0) 0x7fa1640bdea0 con 0x7fa164006e80 2024-03-29T11:28:20.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1627fc640 1 -- 172.21.15.135:0/1105143655 >> v1:172.21.15.135:6790/0 conn(0x7fa1640c9630 legacy=0x7fa164160480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1627fc640 1 -- 172.21.15.135:0/1105143655 >> v1:172.21.15.194:6789/0 conn(0x7fa164008580 legacy=0x7fa16416cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1627fc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:20.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1627fc640 1 monclient: found mon.a 2024-03-29T11:28:20.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:20.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1627fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:28:20.353 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1627fc640 1 -- 172.21.15.135:0/1105143655 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1640cec40 con 0x7fa164006e80 2024-03-29T11:28:20.353 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1627fc640 10 monclient: _finish_auth 0 2024-03-29T11:28:20.353 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1627fc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:20.353 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:20.353 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1729a0640 5 monclient: authenticate success, global_id 4377 2024-03-29T11:28:20.353 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1729a0640 10 monclient: _renew_subs 2024-03-29T11:28:20.353 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1729a0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:28:20.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1729a0640 1 -- 172.21.15.135:0/1105143655 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1640bc2c0 con 0x7fa164006e80 2024-03-29T11:28:20.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1627fc640 1 -- 172.21.15.135:0/1105143655 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa15c004bf0 con 0x7fa164006e80 2024-03-29T11:28:20.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1729a0640 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:20.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1729a0640 10 monclient: _renew_subs 2024-03-29T11:28:20.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1729a0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:28:20.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1729a0640 1 -- 172.21.15.135:0/1105143655 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa16416d250 con 0x7fa164006e80 2024-03-29T11:28:20.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1729a0640 1 librados: init done 2024-03-29T11:28:20.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.356+0000 7fa1627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:20.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.357+0000 7fa1627fc640 1 -- 172.21.15.135:0/1105143655 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fa15c001d80 con 0x7fa164006e80 2024-03-29T11:28:20.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.357+0000 7fa1627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:20.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.357+0000 7fa1627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:20.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.357+0000 7fa1627fc640 10 monclient: dump: 2024-03-29T11:28:20.355 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:20.355 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:20.355 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:20.355 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:20.355 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:20.355 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:20.355 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:20.355 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:20.355 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:20.355 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:20.355 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.357+0000 7fa163fff640 4 set_mon_vals no callback set 2024-03-29T11:28:20.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.359+0000 7fa1627fc640 1 -- 172.21.15.135:0/1105143655 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fa15c01ad70 con 0x7fa164006e80 2024-03-29T11:28:20.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.360+0000 7fa1627fc640 1 -- 172.21.15.135:0/1105143655 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7fa15c084cc0 con 0x7fa164006e80 2024-03-29T11:28:20.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.360+0000 7fa1627fc640 10 client.4377.objecter ms_dispatch 0x7fa16c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:20.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.360+0000 7fa1627fc640 3 client.4377.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:20.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.360+0000 7fa1627fc640 3 client.4377.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:20.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.360+0000 7fa1627fc640 20 client.4377.objecter dump_active .. 0 homeless 2024-03-29T11:28:20.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.360+0000 7fa14ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:20.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.360+0000 7fa14ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:20.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.360+0000 7fa14ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:28:20.358 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.360+0000 7fa14ffff640 1 -- 172.21.15.135:0/1105143655 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa148005080 con 0x7fa164006e80 2024-03-29T11:28:20.358 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.363+0000 7fa17119d640 10 monclient: get_auth_request con 0x7fa15006df20 auth_method 0 2024-03-29T11:28:20.358 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.363+0000 7fa1627fc640 10 client.4377.objecter ms_handle_connect 0x7fa15006df20 2024-03-29T11:28:20.358 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.369+0000 7fa1627fc640 1 -- 172.21.15.135:0/1105143655 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa15c0505a0 con 0x7fa164006e80 2024-03-29T11:28:20.359 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.369+0000 7fa1627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:20.359 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.369+0000 7fa1627fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:20.670 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.679+0000 7fa14ffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:20.670 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.679+0000 7fa14ffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:20.670 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.679+0000 7fa14ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:28:20.670 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.679+0000 7fa14ffff640 1 -- 172.21.15.135:0/1105143655 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa148005330 con 0x7fa164006e80 2024-03-29T11:28:20.670 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.680+0000 7fa1627fc640 1 -- 172.21.15.135:0/1105143655 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7fa15c0524d0 con 0x7fa164006e80 2024-03-29T11:28:20.670 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.680+0000 7fa1627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:20.671 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.680+0000 7fa1627fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:20.671 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:28:20.671 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:20.674 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.684+0000 7fa14ffff640 10 librados: watch_flush enter 2024-03-29T11:28:20.674 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.684+0000 7fa14ffff640 10 librados: watch_flush exit 2024-03-29T11:28:20.674 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.684+0000 7fa14ffff640 20 client.4377.objecter shutdown clearing up homeless session... 2024-03-29T11:28:20.675 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.684+0000 7fa14ffff640 10 client.4377.objecter successfully canceled tick 2024-03-29T11:28:20.675 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.684+0000 7fa14ffff640 1 -- 172.21.15.135:0/1105143655 >> v1:172.21.15.135:6816/3469875285 conn(0x7fa15006df20 legacy=0x7fa1500703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.675 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.684+0000 7fa14ffff640 10 monclient: shutdown 2024-03-29T11:28:20.675 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.684+0000 7fa14ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:20.675 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.684+0000 7fa14ffff640 1 -- 172.21.15.135:0/1105143655 >> v1:172.21.15.135:6789/0 conn(0x7fa164006e80 legacy=0x7fa1640cd1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:20.675 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.685+0000 7fa14ffff640 1 -- 172.21.15.135:0/1105143655 shutdown_connections 2024-03-29T11:28:20.675 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.685+0000 7fa14ffff640 1 -- 172.21.15.135:0/1105143655 >> 172.21.15.135:0/1105143655 conn(0x7fa164005d00 msgr2=0x7fa1640060e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:20.677 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.687+0000 7fa14ffff640 1 -- 172.21.15.135:0/1105143655 shutdown_connections 2024-03-29T11:28:20.677 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.687+0000 7fa14ffff640 1 -- 172.21.15.135:0/1105143655 wait complete. 2024-03-29T11:28:20.677 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:20.687+0000 7fa14ffff640 1 librados: shutdown 2024-03-29T11:28:20.695 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:27.501 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:27.654 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.664+0000 7fa2e21a8640 10 monclient: get_monmap_and_config 2024-03-29T11:28:27.654 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.664+0000 7fa2e21a8640 10 monclient: build_initial_monmap 2024-03-29T11:28:27.654 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.664+0000 7fa2e21a8640 10 monclient: monmap: 2024-03-29T11:28:27.654 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:28:27.654 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:27.654 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:28:27.665233+0000 2024-03-29T11:28:27.654 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:28:27.665233+0000 2024-03-29T11:28:27.655 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:27.655 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:27.655 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:28:27.655 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:28:27.655 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:28:27.655 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:27.656 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.665+0000 7fa2e21a8640 1 Processor -- start 2024-03-29T11:28:27.656 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.665+0000 7fa2e21a8640 1 -- start start 2024-03-29T11:28:27.656 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.665+0000 7fa2e21a8640 10 monclient: init 2024-03-29T11:28:27.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.666+0000 7fa2e21a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:27.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.666+0000 7fa2e21a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:27.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.666+0000 7fa2e21a8640 10 monclient(hunting): picked mon.noname-c con 0x7fa2dc1457f0 addr v1:172.21.15.194:6789/0 2024-03-29T11:28:27.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.666+0000 7fa2e21a8640 10 monclient(hunting): picked mon.noname-b con 0x7fa2dc13f650 addr v1:172.21.15.135:6790/0 2024-03-29T11:28:27.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.666+0000 7fa2e21a8640 10 monclient(hunting): picked mon.noname-a con 0x7fa2dc0a9d70 addr v1:172.21.15.135:6789/0 2024-03-29T11:28:27.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.666+0000 7fa2e21a8640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2dc0aaef0 con 0x7fa2dc0a9d70 2024-03-29T11:28:27.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.666+0000 7fa2e21a8640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2dc145bd0 con 0x7fa2dc1457f0 2024-03-29T11:28:27.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.666+0000 7fa2e21a8640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2dc145eb0 con 0x7fa2dc13f650 2024-03-29T11:28:27.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.666+0000 7fa2e21a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:27.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.666+0000 7fa2e21a8640 10 monclient(hunting): authenticate will time out at 1015.034846s 2024-03-29T11:28:27.663 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.674+0000 7fa2daffd640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7fa2dc13f650 0x7fa2dc0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:42713/0 (socket says 172.21.15.135:42713) 2024-03-29T11:28:27.663 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.674+0000 7fa2daffd640 1 -- 172.21.15.135:0/3144471530 learned_addr learned my addr 172.21.15.135:0/3144471530 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:28:27.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.674+0000 7fa2da7fc640 1 -- 172.21.15.135:0/3144471530 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1133224124 0 0) 0x7fa2dc145eb0 con 0x7fa2dc13f650 2024-03-29T11:28:27.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.674+0000 7fa2da7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:27.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.674+0000 7fa2da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:27.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.674+0000 7fa2da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:27.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.674+0000 7fa2da7fc640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:28:27.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.674+0000 7fa2da7fc640 1 -- 172.21.15.135:0/3144471530 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2c4003520 con 0x7fa2dc13f650 2024-03-29T11:28:27.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.674+0000 7fa2da7fc640 1 -- 172.21.15.135:0/3144471530 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95067522 0 0) 0x7fa2dc0aaef0 con 0x7fa2dc0a9d70 2024-03-29T11:28:27.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.674+0000 7fa2da7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:27.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.674+0000 7fa2da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:27.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.674+0000 7fa2da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:27.665 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.674+0000 7fa2da7fc640 10 monclient(hunting): my global_id is 4380 2024-03-29T11:28:27.665 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.674+0000 7fa2da7fc640 1 -- 172.21.15.135:0/3144471530 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2dc145eb0 con 0x7fa2dc0a9d70 2024-03-29T11:28:27.665 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.675+0000 7fa2da7fc640 1 -- 172.21.15.135:0/3144471530 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 142720133 0 0) 0x7fa2c4003520 con 0x7fa2dc13f650 2024-03-29T11:28:27.665 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.675+0000 7fa2da7fc640 1 -- 172.21.15.135:0/3144471530 >> v1:172.21.15.194:6789/0 conn(0x7fa2dc1457f0 legacy=0x7fa2dc13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:27.667 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.675+0000 7fa2da7fc640 1 -- 172.21.15.135:0/3144471530 >> v1:172.21.15.135:6789/0 conn(0x7fa2dc0a9d70 legacy=0x7fa2dc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.667 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.675+0000 7fa2da7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:27.667 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.675+0000 7fa2da7fc640 1 monclient: found mon.noname-b 2024-03-29T11:28:27.667 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.675+0000 7fa2da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:27.667 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.675+0000 7fa2da7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:28:27.667 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.675+0000 7fa2da7fc640 1 -- 172.21.15.135:0/3144471530 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2dc146340 con 0x7fa2dc13f650 2024-03-29T11:28:27.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.675+0000 7fa2da7fc640 10 monclient: _finish_auth 0 2024-03-29T11:28:27.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.675+0000 7fa2da7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:27.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.675+0000 7fa2da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:27.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.675+0000 7fa2da7fc640 1 -- 172.21.15.135:0/3144471530 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fa2d0001690 con 0x7fa2dc13f650 2024-03-29T11:28:27.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.675+0000 7fa2da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:27.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.675+0000 7fa2da7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:27.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.675+0000 7fa2da7fc640 10 monclient: dump: 2024-03-29T11:28:27.668 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:27.668 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:27.668 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:27.668 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:27.668 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:27.668 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:27.668 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:27.669 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:27.669 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:27.669 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:27.669 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.676+0000 7fa2da7fc640 1 -- 172.21.15.135:0/3144471530 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2d00017f0 con 0x7fa2dc13f650 2024-03-29T11:28:27.669 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.676+0000 7fa2da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:27.669 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.676+0000 7fa2da7fc640 1 -- 172.21.15.135:0/3144471530 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fa2d0001c40 con 0x7fa2dc13f650 2024-03-29T11:28:27.669 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.676+0000 7fa2da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:27.669 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.676+0000 7fa2da7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:27.669 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.676+0000 7fa2da7fc640 10 monclient: dump: 2024-03-29T11:28:27.669 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:27.669 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:27.669 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:27.669 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:27.669 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:27.670 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:27.670 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:27.670 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:27.670 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:27.670 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:27.670 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.676+0000 7fa2e21a8640 5 monclient: authenticate success, global_id 4370 2024-03-29T11:28:27.670 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.676+0000 7fa2e21a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:27.670 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.676+0000 7fa2e21a8640 4 set_mon_vals no callback set 2024-03-29T11:28:27.670 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.677+0000 7fa2e21a8640 10 monclient: shutdown 2024-03-29T11:28:27.670 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.677+0000 7fa2e21a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:27.670 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.677+0000 7fa2e21a8640 1 -- 172.21.15.135:0/3144471530 >> v1:172.21.15.135:6790/0 conn(0x7fa2dc13f650 legacy=0x7fa2dc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.671 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.680+0000 7fa2e21a8640 1 -- 172.21.15.135:0/3144471530 shutdown_connections 2024-03-29T11:28:27.671 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.680+0000 7fa2e21a8640 1 -- 172.21.15.135:0/3144471530 >> 172.21.15.135:0/3144471530 conn(0x7fa2dc066a00 msgr2=0x7fa2dc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:27.671 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.680+0000 7fa2e21a8640 1 -- 172.21.15.135:0/3144471530 shutdown_connections 2024-03-29T11:28:27.671 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.680+0000 7fa2e21a8640 1 -- 172.21.15.135:0/3144471530 wait complete. 2024-03-29T11:28:27.671 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.681+0000 7fa2e21a8640 10 monclient: build_initial_monmap 2024-03-29T11:28:27.671 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.681+0000 7fa2e21a8640 10 monclient: monmap: 2024-03-29T11:28:27.672 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:28:27.672 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:27.672 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:28:27.672 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:28:27.672 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:27.672 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:27.672 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:28:27.672 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:28:27.672 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:28:27.673 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:27.673 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.682+0000 7fa2e21a8640 1 librados: starting msgr at 2024-03-29T11:28:27.673 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.682+0000 7fa2e21a8640 1 librados: starting objecter 2024-03-29T11:28:27.673 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.682+0000 7fa2e21a8640 1 Processor -- start 2024-03-29T11:28:27.673 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.682+0000 7fa2e21a8640 1 -- start start 2024-03-29T11:28:27.673 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.682+0000 7fa2e21a8640 1 librados: setting wanted keys 2024-03-29T11:28:27.674 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.682+0000 7fa2e21a8640 1 librados: calling monclient init 2024-03-29T11:28:27.674 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.682+0000 7fa2e21a8640 10 monclient: init 2024-03-29T11:28:27.674 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.683+0000 7fa2e21a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:27.674 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.683+0000 7fa2e21a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:27.674 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.683+0000 7fa2e21a8640 10 monclient(hunting): picked mon.noname-c con 0x7fa2dc0a9d70 addr v1:172.21.15.135:6790/0 2024-03-29T11:28:27.674 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.683+0000 7fa2e21a8640 10 monclient(hunting): picked mon.noname-b con 0x7fa2dc13f650 addr v1:172.21.15.194:6789/0 2024-03-29T11:28:27.674 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.683+0000 7fa2e21a8640 10 monclient(hunting): picked mon.noname-a con 0x7fa2dc1457f0 addr v1:172.21.15.135:6789/0 2024-03-29T11:28:27.674 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.683+0000 7fa2e21a8640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2dc136960 con 0x7fa2dc1457f0 2024-03-29T11:28:27.674 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.683+0000 7fa2e21a8640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2dc1424f0 con 0x7fa2dc13f650 2024-03-29T11:28:27.674 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.683+0000 7fa2e21a8640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2dc1421a0 con 0x7fa2dc0a9d70 2024-03-29T11:28:27.674 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.683+0000 7fa2e21a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:27.674 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.683+0000 7fa2e21a8640 10 monclient(hunting): authenticate will time out at 1015.051507s 2024-03-29T11:28:27.681 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.691+0000 7fa2daffd640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fa2dc13f650 0x7fa2dc1340d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:47699/0 (socket says 172.21.15.135:47699) 2024-03-29T11:28:27.682 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.691+0000 7fa2daffd640 1 -- 172.21.15.135:0/2153294469 learned_addr learned my addr 172.21.15.135:0/2153294469 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:28:27.682 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.692+0000 7fa2c3fff640 10 client.?.objecter ms_handle_connect 0x7fa2dc13f650 2024-03-29T11:28:27.682 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.692+0000 7fa2c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:27.682 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.692+0000 7fa2c3fff640 1 -- 172.21.15.135:0/2153294469 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948186550 0 0) 0x7fa2dc1424f0 con 0x7fa2dc13f650 2024-03-29T11:28:27.682 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.692+0000 7fa2c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:27.682 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.692+0000 7fa2c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:27.682 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.692+0000 7fa2c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:27.682 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.692+0000 7fa2c3fff640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:28:27.682 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.692+0000 7fa2c3fff640 1 -- 172.21.15.135:0/2153294469 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2c8004eb0 con 0x7fa2dc13f650 2024-03-29T11:28:27.684 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.693+0000 7fa2c3fff640 10 client.?.objecter ms_handle_connect 0x7fa2dc0a9d70 2024-03-29T11:28:27.684 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.693+0000 7fa2c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:27.684 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.693+0000 7fa2c3fff640 1 -- 172.21.15.135:0/2153294469 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269303555 0 0) 0x7fa2dc1421a0 con 0x7fa2dc0a9d70 2024-03-29T11:28:27.684 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.693+0000 7fa2c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:27.684 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.693+0000 7fa2c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:27.684 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.693+0000 7fa2c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:27.685 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.693+0000 7fa2c3fff640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:28:27.685 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.693+0000 7fa2c3fff640 1 -- 172.21.15.135:0/2153294469 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2dc1424f0 con 0x7fa2dc0a9d70 2024-03-29T11:28:27.685 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.693+0000 7fa2c3fff640 1 -- 172.21.15.135:0/2153294469 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3715728779 0 0) 0x7fa2c8004eb0 con 0x7fa2dc13f650 2024-03-29T11:28:27.685 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.693+0000 7fa2c3fff640 1 -- 172.21.15.135:0/2153294469 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2dc1421a0 con 0x7fa2dc13f650 2024-03-29T11:28:27.685 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.693+0000 7fa2c3fff640 1 -- 172.21.15.135:0/2153294469 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fa2d0001d80 con 0x7fa2dc13f650 2024-03-29T11:28:27.685 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.693+0000 7fa2c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:27.685 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.693+0000 7fa2c3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:27.685 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.693+0000 7fa2c3fff640 10 monclient(hunting): dump: 2024-03-29T11:28:27.685 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:27.685 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:27.685 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:27.685 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:27.685 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:27.685 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:27.686 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:27.686 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:27.686 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:27.686 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:27.686 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.693+0000 7fa2c3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:27.686 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.693+0000 7fa2c3fff640 10 client.?.objecter ms_handle_connect 0x7fa2dc1457f0 2024-03-29T11:28:27.686 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.693+0000 7fa2c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:27.686 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 1 -- 172.21.15.135:0/2153294469 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3564649089 0 0) 0x7fa2dc136960 con 0x7fa2dc1457f0 2024-03-29T11:28:27.686 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:27.686 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:27.686 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:27.686 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 10 monclient(hunting): my global_id is 4383 2024-03-29T11:28:27.686 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 1 -- 172.21.15.135:0/2153294469 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2c8004eb0 con 0x7fa2dc1457f0 2024-03-29T11:28:27.686 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 1 -- 172.21.15.135:0/2153294469 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 981643830 0 0) 0x7fa2dc1424f0 con 0x7fa2dc0a9d70 2024-03-29T11:28:27.687 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 1 -- 172.21.15.135:0/2153294469 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2dc136960 con 0x7fa2dc0a9d70 2024-03-29T11:28:27.687 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 1 -- 172.21.15.135:0/2153294469 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fa2d4001b10 con 0x7fa2dc0a9d70 2024-03-29T11:28:27.687 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:27.687 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:27.687 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 10 monclient(hunting): dump: 2024-03-29T11:28:27.687 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:27.687 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:27.687 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:27.687 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:27.687 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:27.687 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:27.687 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:27.687 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:27.687 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:27.688 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:27.688 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 1 -- 172.21.15.135:0/2153294469 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 164933219 0 0) 0x7fa2c8004eb0 con 0x7fa2dc1457f0 2024-03-29T11:28:27.688 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 1 -- 172.21.15.135:0/2153294469 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2dc1424f0 con 0x7fa2dc1457f0 2024-03-29T11:28:27.688 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 1 -- 172.21.15.135:0/2153294469 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fa2cc0013b0 con 0x7fa2dc1457f0 2024-03-29T11:28:27.688 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:27.688 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:27.688 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 10 monclient(hunting): dump: 2024-03-29T11:28:27.688 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:27.688 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:27.688 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:27.688 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:27.688 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:27.688 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:27.688 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:27.688 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:27.689 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:27.689 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:27.689 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 1 -- 172.21.15.135:0/2153294469 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2871174825 0 0) 0x7fa2dc1421a0 con 0x7fa2dc13f650 2024-03-29T11:28:27.689 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 1 -- 172.21.15.135:0/2153294469 >> v1:172.21.15.135:6790/0 conn(0x7fa2dc0a9d70 legacy=0x7fa2dc13fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.689 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 1 -- 172.21.15.135:0/2153294469 >> v1:172.21.15.135:6789/0 conn(0x7fa2dc1457f0 legacy=0x7fa2dc1347e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.689 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:27.689 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 1 monclient: found mon.b 2024-03-29T11:28:27.689 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:27.689 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:28:27.689 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 1 -- 172.21.15.135:0/2153294469 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2dc144d10 con 0x7fa2dc13f650 2024-03-29T11:28:27.689 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 10 monclient: _finish_auth 0 2024-03-29T11:28:27.689 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:27.689 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:27.689 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2e21a8640 5 monclient: authenticate success, global_id 4375 2024-03-29T11:28:27.689 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2e21a8640 10 monclient: _renew_subs 2024-03-29T11:28:27.690 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2e21a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:28:27.690 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.694+0000 7fa2e21a8640 1 -- 172.21.15.135:0/2153294469 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2dc141b50 con 0x7fa2dc13f650 2024-03-29T11:28:27.690 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.695+0000 7fa2e21a8640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:27.690 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.695+0000 7fa2e21a8640 10 monclient: _renew_subs 2024-03-29T11:28:27.690 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.695+0000 7fa2e21a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:28:27.690 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.695+0000 7fa2e21a8640 1 -- 172.21.15.135:0/2153294469 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2dc142a10 con 0x7fa2dc13f650 2024-03-29T11:28:27.690 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.695+0000 7fa2e21a8640 1 librados: init done 2024-03-29T11:28:27.691 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.696+0000 7fa2c1ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:27.691 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.696+0000 7fa2c1ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:27.691 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.696+0000 7fa2c1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:28:27.691 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.696+0000 7fa2c1ffb640 1 -- 172.21.15.135:0/2153294469 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2dc142330 con 0x7fa2dc13f650 2024-03-29T11:28:27.691 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.698+0000 7fa2c3fff640 1 -- 172.21.15.135:0/2153294469 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2d0001670 con 0x7fa2dc13f650 2024-03-29T11:28:27.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.698+0000 7fa2c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:27.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.698+0000 7fa2c3fff640 1 -- 172.21.15.135:0/2153294469 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fa2d0004d90 con 0x7fa2dc13f650 2024-03-29T11:28:27.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.698+0000 7fa2c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:27.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.698+0000 7fa2c3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:27.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.698+0000 7fa2c3fff640 10 monclient: dump: 2024-03-29T11:28:27.692 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:27.692 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:27.692 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:27.692 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:27.692 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:27.692 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:27.692 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:27.692 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:27.692 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:27.692 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:27.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.699+0000 7fa2d97fa640 4 set_mon_vals no callback set 2024-03-29T11:28:27.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.700+0000 7fa2c3fff640 1 -- 172.21.15.135:0/2153294469 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fa2d001ad60 con 0x7fa2dc13f650 2024-03-29T11:28:27.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.700+0000 7fa2c3fff640 1 -- 172.21.15.135:0/2153294469 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7fa2d0084bb0 con 0x7fa2dc13f650 2024-03-29T11:28:27.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.701+0000 7fa2db7fe640 10 monclient: get_auth_request con 0x7fa2c806e020 auth_method 0 2024-03-29T11:28:27.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.701+0000 7fa2c3fff640 10 client.4375.objecter ms_dispatch 0x7fa2dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:27.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.701+0000 7fa2c3fff640 3 client.4375.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:27.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.701+0000 7fa2c3fff640 3 client.4375.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:27.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.701+0000 7fa2c3fff640 20 client.4375.objecter dump_active .. 0 homeless 2024-03-29T11:28:27.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.701+0000 7fa2c3fff640 10 client.4375.objecter ms_handle_connect 0x7fa2c806e020 2024-03-29T11:28:27.698 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.707+0000 7fa2c3fff640 1 -- 172.21.15.135:0/2153294469 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa2d0050490 con 0x7fa2dc13f650 2024-03-29T11:28:27.698 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.707+0000 7fa2c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:27.698 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:27.708+0000 7fa2c3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:28.020 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:28.029+0000 7fa2c1ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:28.020 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:28.029+0000 7fa2c1ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:28.020 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:28.029+0000 7fa2c1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:28:28.020 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:28.029+0000 7fa2c1ffb640 1 -- 172.21.15.135:0/2153294469 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2dc075f10 con 0x7fa2dc13f650 2024-03-29T11:28:28.020 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:28.031+0000 7fa2c3fff640 1 -- 172.21.15.135:0/2153294469 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7fa2d00523c0 con 0x7fa2dc13f650 2024-03-29T11:28:28.020 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:28.031+0000 7fa2c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:28.020 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:28.031+0000 7fa2c3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:28.022 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:28:28.023 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:28.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:28.035+0000 7fa2c1ffb640 10 librados: watch_flush enter 2024-03-29T11:28:28.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:28.035+0000 7fa2c1ffb640 10 librados: watch_flush exit 2024-03-29T11:28:28.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:28.035+0000 7fa2c1ffb640 20 client.4375.objecter shutdown clearing up homeless session... 2024-03-29T11:28:28.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:28.035+0000 7fa2c1ffb640 10 client.4375.objecter successfully canceled tick 2024-03-29T11:28:28.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:28.036+0000 7fa2c1ffb640 1 -- 172.21.15.135:0/2153294469 >> v1:172.21.15.135:6816/3469875285 conn(0x7fa2c806e020 legacy=0x7fa2c80704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:28.026 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:28.036+0000 7fa2c1ffb640 10 monclient: shutdown 2024-03-29T11:28:28.026 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:28.036+0000 7fa2c1ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:28.026 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:28.036+0000 7fa2c1ffb640 1 -- 172.21.15.135:0/2153294469 >> v1:172.21.15.194:6789/0 conn(0x7fa2dc13f650 legacy=0x7fa2dc1340d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:28.026 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:28.036+0000 7fa2c1ffb640 1 -- 172.21.15.135:0/2153294469 shutdown_connections 2024-03-29T11:28:28.026 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:28.036+0000 7fa2c1ffb640 1 -- 172.21.15.135:0/2153294469 >> 172.21.15.135:0/2153294469 conn(0x7fa2dc066a00 msgr2=0x7fa2dc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:28.026 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:28.036+0000 7fa2c1ffb640 1 -- 172.21.15.135:0/2153294469 shutdown_connections 2024-03-29T11:28:28.026 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:28.036+0000 7fa2c1ffb640 1 -- 172.21.15.135:0/2153294469 wait complete. 2024-03-29T11:28:28.026 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:28.037+0000 7fa2c1ffb640 1 librados: shutdown 2024-03-29T11:28:28.044 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:34.850 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:35.004 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.015+0000 7f45e118f640 10 monclient: get_monmap_and_config 2024-03-29T11:28:35.005 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.015+0000 7f45e118f640 10 monclient: build_initial_monmap 2024-03-29T11:28:35.005 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.015+0000 7f45e118f640 10 monclient: monmap: 2024-03-29T11:28:35.005 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:28:35.005 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:35.005 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:28:35.015786+0000 2024-03-29T11:28:35.005 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:28:35.015786+0000 2024-03-29T11:28:35.005 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:35.005 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:35.005 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:28:35.005 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:28:35.005 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:28:35.005 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:35.006 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.016+0000 7f45e118f640 1 Processor -- start 2024-03-29T11:28:35.006 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.016+0000 7f45e118f640 1 -- start start 2024-03-29T11:28:35.006 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.016+0000 7f45e118f640 10 monclient: init 2024-03-29T11:28:35.007 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.018+0000 7f45e118f640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:35.007 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.018+0000 7f45e118f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:35.007 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.018+0000 7f45e118f640 10 monclient(hunting): picked mon.noname-c con 0x7f45dc1457f0 addr v1:172.21.15.194:6789/0 2024-03-29T11:28:35.007 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.018+0000 7f45e118f640 10 monclient(hunting): picked mon.noname-b con 0x7f45dc13f650 addr v1:172.21.15.135:6790/0 2024-03-29T11:28:35.007 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.018+0000 7f45e118f640 10 monclient(hunting): picked mon.noname-a con 0x7f45dc0a9d70 addr v1:172.21.15.135:6789/0 2024-03-29T11:28:35.008 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.018+0000 7f45e118f640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45dc0aaef0 con 0x7f45dc0a9d70 2024-03-29T11:28:35.008 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.018+0000 7f45e118f640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45dc145bd0 con 0x7f45dc1457f0 2024-03-29T11:28:35.008 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.018+0000 7f45e118f640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45dc145eb0 con 0x7f45dc13f650 2024-03-29T11:28:35.008 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.018+0000 7f45e118f640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:35.008 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.018+0000 7f45e118f640 10 monclient(hunting): authenticate will time out at 1022.386309s 2024-03-29T11:28:35.008 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.018+0000 7f45dad9d640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f45dc1457f0 0x7f45dc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:57726/0 (socket says 172.21.15.135:57726) 2024-03-29T11:28:35.008 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.018+0000 7f45dad9d640 1 -- 172.21.15.135:0/3719368441 learned_addr learned my addr 172.21.15.135:0/3719368441 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:28:35.011 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.018+0000 7f45d9d9b640 1 -- 172.21.15.135:0/3719368441 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482785607 0 0) 0x7f45dc145bd0 con 0x7f45dc1457f0 2024-03-29T11:28:35.011 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.018+0000 7f45d9d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:28:35.011 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.018+0000 7f45d9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:35.011 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.018+0000 7f45d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:35.011 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.018+0000 7f45d9d9b640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:28:35.011 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.018+0000 7f45d9d9b640 1 -- 172.21.15.135:0/3719368441 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45c4003520 con 0x7f45dc1457f0 2024-03-29T11:28:35.011 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45d9d9b640 1 -- 172.21.15.135:0/3719368441 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622726064 0 0) 0x7f45dc145eb0 con 0x7f45dc13f650 2024-03-29T11:28:35.011 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45d9d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:28:35.011 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45d9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:35.011 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:35.012 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45d9d9b640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:28:35.012 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45d9d9b640 1 -- 172.21.15.135:0/3719368441 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45dc145bd0 con 0x7f45dc13f650 2024-03-29T11:28:35.012 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45d9d9b640 1 -- 172.21.15.135:0/3719368441 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2123907502 0 0) 0x7f45dc0aaef0 con 0x7f45dc0a9d70 2024-03-29T11:28:35.012 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45d9d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:28:35.012 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45d9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:35.012 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:35.012 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45d9d9b640 10 monclient(hunting): my global_id is 4386 2024-03-29T11:28:35.012 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45d9d9b640 1 -- 172.21.15.135:0/3719368441 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45dc145eb0 con 0x7f45dc0a9d70 2024-03-29T11:28:35.012 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45d9d9b640 1 -- 172.21.15.135:0/3719368441 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1695596012 0 0) 0x7f45c4003520 con 0x7f45dc1457f0 2024-03-29T11:28:35.012 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45d9d9b640 1 -- 172.21.15.135:0/3719368441 >> v1:172.21.15.135:6790/0 conn(0x7f45dc13f650 legacy=0x7f45dc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.012 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45d9d9b640 1 -- 172.21.15.135:0/3719368441 >> v1:172.21.15.135:6789/0 conn(0x7f45dc0a9d70 legacy=0x7f45dc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.012 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45d9d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:28:35.012 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45d9d9b640 1 monclient: found mon.noname-c 2024-03-29T11:28:35.012 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45d9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:35.013 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45d9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:28:35.013 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45d9d9b640 1 -- 172.21.15.135:0/3719368441 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45dc146340 con 0x7f45dc1457f0 2024-03-29T11:28:35.013 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45d9d9b640 10 monclient: _finish_auth 0 2024-03-29T11:28:35.013 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45d9d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:28:35.013 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45d9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:35.013 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45e118f640 5 monclient: authenticate success, global_id 4378 2024-03-29T11:28:35.013 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45e118f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:35.013 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45d9d9b640 1 -- 172.21.15.135:0/3719368441 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f45cc001690 con 0x7f45dc1457f0 2024-03-29T11:28:35.013 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45d9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:35.013 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.019+0000 7f45d9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:35.013 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.020+0000 7f45d9d9b640 10 monclient: dump: 2024-03-29T11:28:35.013 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:35.013 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:35.013 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:35.014 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:35.014 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:35.014 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:35.014 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:35.014 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:35.014 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:35.014 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:35.014 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.020+0000 7f45e118f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:35.014 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.020+0000 7f45d9d9b640 1 -- 172.21.15.135:0/3719368441 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45cc0017f0 con 0x7f45dc1457f0 2024-03-29T11:28:35.014 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.020+0000 7f45d9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:35.014 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.020+0000 7f45d9d9b640 1 -- 172.21.15.135:0/3719368441 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f45cc001c40 con 0x7f45dc1457f0 2024-03-29T11:28:35.014 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.020+0000 7f45e118f640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:35.014 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.020+0000 7f45e118f640 4 set_mon_vals no callback set 2024-03-29T11:28:35.014 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.021+0000 7f45d9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:35.015 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.021+0000 7f45d9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:35.015 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.021+0000 7f45d9d9b640 10 monclient: dump: 2024-03-29T11:28:35.015 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:35.015 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:35.015 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:35.015 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:35.015 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:35.015 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:35.015 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:35.015 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:35.015 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:35.015 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:35.015 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.021+0000 7f45e118f640 10 monclient: shutdown 2024-03-29T11:28:35.015 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.021+0000 7f45e118f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:35.015 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.021+0000 7f45e118f640 1 -- 172.21.15.135:0/3719368441 >> v1:172.21.15.194:6789/0 conn(0x7f45dc1457f0 legacy=0x7f45dc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.016 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.021+0000 7f45e118f640 1 -- 172.21.15.135:0/3719368441 shutdown_connections 2024-03-29T11:28:35.016 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.021+0000 7f45e118f640 1 -- 172.21.15.135:0/3719368441 >> 172.21.15.135:0/3719368441 conn(0x7f45dc066a00 msgr2=0x7f45dc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:35.016 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.022+0000 7f45e118f640 1 -- 172.21.15.135:0/3719368441 shutdown_connections 2024-03-29T11:28:35.021 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.022+0000 7f45e118f640 1 -- 172.21.15.135:0/3719368441 wait complete. 2024-03-29T11:28:35.021 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.023+0000 7f45e118f640 10 monclient: build_initial_monmap 2024-03-29T11:28:35.022 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.023+0000 7f45e118f640 10 monclient: monmap: 2024-03-29T11:28:35.022 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:28:35.022 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:35.022 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:28:35.022 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:28:35.022 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:35.022 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:35.022 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:28:35.022 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:28:35.022 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:28:35.022 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:35.022 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.024+0000 7f45e118f640 1 librados: starting msgr at 2024-03-29T11:28:35.023 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.024+0000 7f45e118f640 1 librados: starting objecter 2024-03-29T11:28:35.023 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.024+0000 7f45e118f640 1 Processor -- start 2024-03-29T11:28:35.023 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.024+0000 7f45e118f640 1 -- start start 2024-03-29T11:28:35.023 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.024+0000 7f45e118f640 1 librados: setting wanted keys 2024-03-29T11:28:35.023 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.024+0000 7f45e118f640 1 librados: calling monclient init 2024-03-29T11:28:35.023 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.024+0000 7f45e118f640 10 monclient: init 2024-03-29T11:28:35.023 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.026+0000 7f45e118f640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:35.023 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.026+0000 7f45e118f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:35.023 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.026+0000 7f45e118f640 10 monclient(hunting): picked mon.noname-a con 0x7f45dc0a9d70 addr v1:172.21.15.135:6789/0 2024-03-29T11:28:35.023 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.026+0000 7f45e118f640 10 monclient(hunting): picked mon.noname-c con 0x7f45dc13f650 addr v1:172.21.15.135:6790/0 2024-03-29T11:28:35.023 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.026+0000 7f45e118f640 10 monclient(hunting): picked mon.noname-b con 0x7f45dc1457f0 addr v1:172.21.15.194:6789/0 2024-03-29T11:28:35.023 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.026+0000 7f45e118f640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45dc0720c0 con 0x7f45dc0a9d70 2024-03-29T11:28:35.023 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.026+0000 7f45dad9d640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f45dc0a9d70 0x7f45dc144a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:56592/0 (socket says 172.21.15.135:56592) 2024-03-29T11:28:35.023 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.026+0000 7f45e118f640 1 -- 172.21.15.135:0/2435586363 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45dc0712d0 con 0x7f45dc1457f0 2024-03-29T11:28:35.024 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.026+0000 7f45dad9d640 1 -- 172.21.15.135:0/2435586363 learned_addr learned my addr 172.21.15.135:0/2435586363 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:28:35.024 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.026+0000 7f45e118f640 1 -- 172.21.15.135:0/2435586363 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45dc142bb0 con 0x7f45dc13f650 2024-03-29T11:28:35.024 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.026+0000 7f45e118f640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:35.024 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.026+0000 7f45e118f640 10 monclient(hunting): authenticate will time out at 1022.394407s 2024-03-29T11:28:35.024 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.026+0000 7f45c37fe640 10 client.?.objecter ms_handle_connect 0x7f45dc1457f0 2024-03-29T11:28:35.024 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.026+0000 7f45c37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:35.024 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.026+0000 7f45c37fe640 10 client.?.objecter ms_handle_connect 0x7f45dc13f650 2024-03-29T11:28:35.024 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.026+0000 7f45c37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:35.024 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.026+0000 7f45c37fe640 10 client.?.objecter ms_handle_connect 0x7f45dc0a9d70 2024-03-29T11:28:35.024 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.026+0000 7f45c37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:35.024 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.026+0000 7f45c37fe640 1 -- 172.21.15.135:0/2435586363 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216215693 0 0) 0x7f45dc0712d0 con 0x7f45dc1457f0 2024-03-29T11:28:35.024 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.026+0000 7f45c37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:35.024 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.026+0000 7f45c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:35.024 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.026+0000 7f45c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:35.024 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.026+0000 7f45c37fe640 10 monclient(hunting): my global_id is 4381 2024-03-29T11:28:35.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.026+0000 7f45c37fe640 1 -- 172.21.15.135:0/2435586363 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45c8004eb0 con 0x7f45dc1457f0 2024-03-29T11:28:35.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.026+0000 7f45c37fe640 1 -- 172.21.15.135:0/2435586363 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2742909156 0 0) 0x7f45dc142bb0 con 0x7f45dc13f650 2024-03-29T11:28:35.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:35.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:35.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:35.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:28:35.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 1 -- 172.21.15.135:0/2435586363 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45dc0712d0 con 0x7f45dc13f650 2024-03-29T11:28:35.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 1 -- 172.21.15.135:0/2435586363 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667093405 0 0) 0x7f45dc0720c0 con 0x7f45dc0a9d70 2024-03-29T11:28:35.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:35.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:35.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:35.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:28:35.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 1 -- 172.21.15.135:0/2435586363 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45dc142bb0 con 0x7f45dc0a9d70 2024-03-29T11:28:35.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 1 -- 172.21.15.135:0/2435586363 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3476212358 0 0) 0x7f45c8004eb0 con 0x7f45dc1457f0 2024-03-29T11:28:35.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 1 -- 172.21.15.135:0/2435586363 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45dc0720c0 con 0x7f45dc1457f0 2024-03-29T11:28:35.026 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 1 -- 172.21.15.135:0/2435586363 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f45d4001cb0 con 0x7f45dc1457f0 2024-03-29T11:28:35.026 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:35.026 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:35.026 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 10 monclient(hunting): dump: 2024-03-29T11:28:35.026 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:35.026 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:35.026 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:35.026 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:35.026 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:35.026 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:35.026 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:35.026 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:35.026 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:35.026 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:35.027 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:35.027 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 1 -- 172.21.15.135:0/2435586363 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4283940531 0 0) 0x7f45dc0712d0 con 0x7f45dc13f650 2024-03-29T11:28:35.027 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 1 -- 172.21.15.135:0/2435586363 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45c8004eb0 con 0x7f45dc13f650 2024-03-29T11:28:35.027 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 1 -- 172.21.15.135:0/2435586363 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3689056682 0 0) 0x7f45dc142bb0 con 0x7f45dc0a9d70 2024-03-29T11:28:35.027 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 1 -- 172.21.15.135:0/2435586363 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45dc0712d0 con 0x7f45dc0a9d70 2024-03-29T11:28:35.027 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 1 -- 172.21.15.135:0/2435586363 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f45d00017e0 con 0x7f45dc13f650 2024-03-29T11:28:35.027 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:35.027 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:35.027 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 10 monclient(hunting): dump: 2024-03-29T11:28:35.027 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:35.027 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:35.027 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:35.027 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:35.027 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:35.027 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:35.028 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:35.028 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:35.028 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:35.028 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:35.028 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.027+0000 7f45c37fe640 1 -- 172.21.15.135:0/2435586363 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f45cc001b00 con 0x7f45dc0a9d70 2024-03-29T11:28:35.028 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.028+0000 7f45c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:35.028 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.028+0000 7f45c37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:35.028 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.028+0000 7f45c37fe640 10 monclient(hunting): dump: 2024-03-29T11:28:35.028 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:35.028 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:35.028 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:35.028 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:35.028 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:35.028 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:35.028 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:35.029 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:35.029 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:35.029 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:35.029 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.028+0000 7f45c37fe640 1 -- 172.21.15.135:0/2435586363 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3426089157 0 0) 0x7f45dc0720c0 con 0x7f45dc1457f0 2024-03-29T11:28:35.029 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.028+0000 7f45c37fe640 1 -- 172.21.15.135:0/2435586363 >> v1:172.21.15.135:6790/0 conn(0x7f45dc13f650 legacy=0x7f45dc1424a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.029 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.028+0000 7f45c37fe640 1 -- 172.21.15.135:0/2435586363 >> v1:172.21.15.135:6789/0 conn(0x7f45dc0a9d70 legacy=0x7f45dc144a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.029 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.028+0000 7f45c37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:28:35.029 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.028+0000 7f45c37fe640 1 monclient: found mon.b 2024-03-29T11:28:35.029 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.028+0000 7f45c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:35.029 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.028+0000 7f45c37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:28:35.029 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.028+0000 7f45c37fe640 1 -- 172.21.15.135:0/2435586363 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45dc06eaa0 con 0x7f45dc1457f0 2024-03-29T11:28:35.029 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.028+0000 7f45c37fe640 10 monclient: _finish_auth 0 2024-03-29T11:28:35.029 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.028+0000 7f45c37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:28:35.029 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.028+0000 7f45c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:35.029 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.028+0000 7f45e118f640 5 monclient: authenticate success, global_id 4381 2024-03-29T11:28:35.030 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.028+0000 7f45e118f640 10 monclient: _renew_subs 2024-03-29T11:28:35.030 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.028+0000 7f45e118f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:28:35.030 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.028+0000 7f45e118f640 1 -- 172.21.15.135:0/2435586363 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45dc06ed80 con 0x7f45dc1457f0 2024-03-29T11:28:35.030 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.028+0000 7f45e118f640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:35.030 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.028+0000 7f45e118f640 10 monclient: _renew_subs 2024-03-29T11:28:35.030 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.028+0000 7f45e118f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:28:35.030 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.028+0000 7f45e118f640 1 -- 172.21.15.135:0/2435586363 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45dc071c30 con 0x7f45dc1457f0 2024-03-29T11:28:35.030 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.028+0000 7f45e118f640 1 librados: init done 2024-03-29T11:28:35.030 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.028+0000 7f45c37fe640 1 -- 172.21.15.135:0/2435586363 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45d4003970 con 0x7f45dc1457f0 2024-03-29T11:28:35.030 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.028+0000 7f45c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:35.030 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.028+0000 7f45d8d99640 4 set_mon_vals no callback set 2024-03-29T11:28:35.030 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.029+0000 7f45c37fe640 1 -- 172.21.15.135:0/2435586363 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f45d4003cc0 con 0x7f45dc1457f0 2024-03-29T11:28:35.030 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.029+0000 7f45c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:35.031 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.029+0000 7f45c37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:35.031 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.029+0000 7f45c37fe640 10 monclient: dump: 2024-03-29T11:28:35.031 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:35.031 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:35.031 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:35.031 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:35.031 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:35.031 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:35.031 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:35.031 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:35.031 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:35.031 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:35.031 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.031+0000 7f45c37fe640 1 -- 172.21.15.135:0/2435586363 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f45d4003e20 con 0x7f45dc1457f0 2024-03-29T11:28:35.031 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.031+0000 7f45e118f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:35.031 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.031+0000 7f45e118f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:35.032 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.031+0000 7f45e118f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:28:35.032 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.031+0000 7f45e118f640 1 -- 172.21.15.135:0/2435586363 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45bc005080 con 0x7f45dc1457f0 2024-03-29T11:28:35.032 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.032+0000 7f45dad9d640 10 monclient: get_auth_request con 0x7f45c806e230 auth_method 0 2024-03-29T11:28:35.032 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.032+0000 7f45c37fe640 1 -- 172.21.15.135:0/2435586363 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f45d4083b90 con 0x7f45dc1457f0 2024-03-29T11:28:35.032 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.032+0000 7f45c37fe640 10 client.4381.objecter ms_dispatch 0x7f45dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:35.032 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.032+0000 7f45c37fe640 3 client.4381.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:35.032 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.032+0000 7f45c37fe640 3 client.4381.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:35.032 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.032+0000 7f45c37fe640 20 client.4381.objecter dump_active .. 0 homeless 2024-03-29T11:28:35.032 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.032+0000 7f45c37fe640 10 client.4381.objecter ms_handle_connect 0x7f45c806e230 2024-03-29T11:28:35.033 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.038+0000 7f45c37fe640 1 -- 172.21.15.135:0/2435586363 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f45d404f470 con 0x7f45dc1457f0 2024-03-29T11:28:35.033 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.038+0000 7f45c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:35.033 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.038+0000 7f45c37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:35.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.416+0000 7f45e118f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:35.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.416+0000 7f45e118f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:35.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.416+0000 7f45e118f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:28:35.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.416+0000 7f45e118f640 1 -- 172.21.15.135:0/2435586363 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45bc005330 con 0x7f45dc1457f0 2024-03-29T11:28:35.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.417+0000 7f45c37fe640 1 -- 172.21.15.135:0/2435586363 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f45d40513a0 con 0x7f45dc1457f0 2024-03-29T11:28:35.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.417+0000 7f45c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:35.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.417+0000 7f45c37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:35.407 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:28:35.407 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:35.411 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.421+0000 7f45e118f640 10 librados: watch_flush enter 2024-03-29T11:28:35.411 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.421+0000 7f45e118f640 10 librados: watch_flush exit 2024-03-29T11:28:35.411 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.421+0000 7f45e118f640 20 client.4381.objecter shutdown clearing up homeless session... 2024-03-29T11:28:35.411 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.421+0000 7f45e118f640 10 client.4381.objecter successfully canceled tick 2024-03-29T11:28:35.411 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.421+0000 7f45e118f640 1 -- 172.21.15.135:0/2435586363 >> v1:172.21.15.135:6816/3469875285 conn(0x7f45c806e230 legacy=0x7f45c80706f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.411 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.421+0000 7f45e118f640 10 monclient: shutdown 2024-03-29T11:28:35.412 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.421+0000 7f45e118f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:35.412 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.421+0000 7f45e118f640 1 -- 172.21.15.135:0/2435586363 >> v1:172.21.15.194:6789/0 conn(0x7f45dc1457f0 legacy=0x7f45dc1e64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:35.412 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.421+0000 7f45e118f640 1 -- 172.21.15.135:0/2435586363 shutdown_connections 2024-03-29T11:28:35.412 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.421+0000 7f45e118f640 1 -- 172.21.15.135:0/2435586363 >> 172.21.15.135:0/2435586363 conn(0x7f45dc066a00 msgr2=0x7f45dc13ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:35.412 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.422+0000 7f45e118f640 1 -- 172.21.15.135:0/2435586363 shutdown_connections 2024-03-29T11:28:35.412 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.422+0000 7f45e118f640 1 -- 172.21.15.135:0/2435586363 wait complete. 2024-03-29T11:28:35.412 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:35.422+0000 7f45e118f640 1 librados: shutdown 2024-03-29T11:28:35.427 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:42.236 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:42.389 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.399+0000 7f3b1818d640 10 monclient: get_monmap_and_config 2024-03-29T11:28:42.389 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.399+0000 7f3b1818d640 10 monclient: build_initial_monmap 2024-03-29T11:28:42.389 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.399+0000 7f3b1818d640 10 monclient: monmap: 2024-03-29T11:28:42.389 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:28:42.389 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:42.389 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:28:42.399864+0000 2024-03-29T11:28:42.390 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:28:42.399864+0000 2024-03-29T11:28:42.390 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:42.390 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:42.390 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:28:42.390 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:28:42.391 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:28:42.391 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:42.391 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.400+0000 7f3b1818d640 1 Processor -- start 2024-03-29T11:28:42.391 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.400+0000 7f3b1818d640 1 -- start start 2024-03-29T11:28:42.392 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.400+0000 7f3b1818d640 10 monclient: init 2024-03-29T11:28:42.392 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.401+0000 7f3b1818d640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:42.392 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.401+0000 7f3b1818d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:42.392 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.401+0000 7f3b1818d640 10 monclient(hunting): picked mon.noname-b con 0x7f3b101457f0 addr v1:172.21.15.135:6790/0 2024-03-29T11:28:42.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.401+0000 7f3b1818d640 10 monclient(hunting): picked mon.noname-c con 0x7f3b1013f650 addr v1:172.21.15.194:6789/0 2024-03-29T11:28:42.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.403+0000 7f3b1818d640 10 monclient(hunting): picked mon.noname-a con 0x7f3b100a9d70 addr v1:172.21.15.135:6789/0 2024-03-29T11:28:42.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.403+0000 7f3b1818d640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b100aaef0 con 0x7f3b100a9d70 2024-03-29T11:28:42.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.403+0000 7f3b1818d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b10145bd0 con 0x7f3b1013f650 2024-03-29T11:28:42.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.403+0000 7f3b1818d640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b10145eb0 con 0x7f3b101457f0 2024-03-29T11:28:42.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.403+0000 7f3b1818d640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:42.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.403+0000 7f3b1818d640 10 monclient(hunting): authenticate will time out at 1029.771435s 2024-03-29T11:28:42.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.402+0000 7f3b15f29640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f3b101457f0 0x7f3b1013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:55612/0 (socket says 172.21.15.135:55612) 2024-03-29T11:28:42.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.403+0000 7f3b15f29640 1 -- 172.21.15.135:0/1329581014 learned_addr learned my addr 172.21.15.135:0/1329581014 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:28:42.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.403+0000 7f3b14f27640 1 -- 172.21.15.135:0/1329581014 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190181620 0 0) 0x7f3b10145eb0 con 0x7f3b101457f0 2024-03-29T11:28:42.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.403+0000 7f3b14f27640 10 monclient(hunting): _negotiate 2024-03-29T11:28:42.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.403+0000 7f3b14f27640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:42.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.403+0000 7f3b14f27640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:42.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.403+0000 7f3b14f27640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:28:42.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.403+0000 7f3b14f27640 1 -- 172.21.15.135:0/1329581014 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3afc003520 con 0x7f3b101457f0 2024-03-29T11:28:42.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.404+0000 7f3b14f27640 1 -- 172.21.15.135:0/1329581014 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2181766330 0 0) 0x7f3afc003520 con 0x7f3b101457f0 2024-03-29T11:28:42.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.404+0000 7f3b14f27640 1 -- 172.21.15.135:0/1329581014 >> v1:172.21.15.194:6789/0 conn(0x7f3b1013f650 legacy=0x7f3b100b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:42.396 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.405+0000 7f3b14f27640 1 -- 172.21.15.135:0/1329581014 >> v1:172.21.15.135:6789/0 conn(0x7f3b100a9d70 legacy=0x7f3b1014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:42.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.409+0000 7f3b14f27640 10 monclient: _finish_hunting 0 2024-03-29T11:28:42.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.409+0000 7f3b14f27640 1 monclient: found mon.noname-b 2024-03-29T11:28:42.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.409+0000 7f3b14f27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:42.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.409+0000 7f3b14f27640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:28:42.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.409+0000 7f3b14f27640 1 -- 172.21.15.135:0/1329581014 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b10146340 con 0x7f3b101457f0 2024-03-29T11:28:42.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.409+0000 7f3b14f27640 10 monclient: _finish_auth 0 2024-03-29T11:28:42.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.409+0000 7f3b14f27640 10 monclient: _check_auth_tickets 2024-03-29T11:28:42.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.409+0000 7f3b14f27640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:42.400 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.409+0000 7f3b14f27640 1 -- 172.21.15.135:0/1329581014 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f3b0c001690 con 0x7f3b101457f0 2024-03-29T11:28:42.400 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.409+0000 7f3b14f27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:42.400 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.409+0000 7f3b14f27640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:42.400 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.409+0000 7f3b14f27640 10 monclient: dump: 2024-03-29T11:28:42.400 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:42.400 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:42.400 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:42.400 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:42.400 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:42.400 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:42.401 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:42.401 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:42.401 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:42.401 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:42.401 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.409+0000 7f3b1818d640 5 monclient: authenticate success, global_id 4382 2024-03-29T11:28:42.401 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.409+0000 7f3b1818d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:42.402 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.409+0000 7f3b14f27640 1 -- 172.21.15.135:0/1329581014 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b0c001860 con 0x7f3b101457f0 2024-03-29T11:28:42.402 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.409+0000 7f3b14f27640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:42.402 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.409+0000 7f3b14f27640 1 -- 172.21.15.135:0/1329581014 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f3b0c001cb0 con 0x7f3b101457f0 2024-03-29T11:28:42.402 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.409+0000 7f3b14f27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:42.402 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.409+0000 7f3b14f27640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:42.403 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.409+0000 7f3b14f27640 10 monclient: dump: 2024-03-29T11:28:42.403 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:42.403 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:42.403 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:42.403 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:42.403 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:42.403 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:42.403 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:42.403 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:42.403 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:42.403 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:42.403 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.409+0000 7f3b1818d640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:42.403 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.410+0000 7f3b1818d640 4 set_mon_vals no callback set 2024-03-29T11:28:42.404 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.410+0000 7f3b1818d640 10 monclient: shutdown 2024-03-29T11:28:42.404 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.411+0000 7f3b1818d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:42.404 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.411+0000 7f3b1818d640 1 -- 172.21.15.135:0/1329581014 >> v1:172.21.15.135:6790/0 conn(0x7f3b101457f0 legacy=0x7f3b1013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:42.408 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.414+0000 7f3b1818d640 1 -- 172.21.15.135:0/1329581014 shutdown_connections 2024-03-29T11:28:42.408 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.414+0000 7f3b1818d640 1 -- 172.21.15.135:0/1329581014 >> 172.21.15.135:0/1329581014 conn(0x7f3b10066a00 msgr2=0x7f3b10066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:42.408 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.414+0000 7f3b1818d640 1 -- 172.21.15.135:0/1329581014 shutdown_connections 2024-03-29T11:28:42.408 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.414+0000 7f3b1818d640 1 -- 172.21.15.135:0/1329581014 wait complete. 2024-03-29T11:28:42.408 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.415+0000 7f3b1818d640 10 monclient: build_initial_monmap 2024-03-29T11:28:42.409 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.416+0000 7f3b1818d640 10 monclient: monmap: 2024-03-29T11:28:42.409 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:28:42.409 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:42.409 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:28:42.409 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:28:42.409 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:42.409 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:42.409 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:28:42.409 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:28:42.410 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:28:42.410 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:42.410 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.416+0000 7f3b1818d640 1 librados: starting msgr at 2024-03-29T11:28:42.410 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.416+0000 7f3b1818d640 1 librados: starting objecter 2024-03-29T11:28:42.410 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.416+0000 7f3b1818d640 1 Processor -- start 2024-03-29T11:28:42.410 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.416+0000 7f3b1818d640 1 -- start start 2024-03-29T11:28:42.410 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.417+0000 7f3b1818d640 1 librados: setting wanted keys 2024-03-29T11:28:42.410 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.417+0000 7f3b1818d640 1 librados: calling monclient init 2024-03-29T11:28:42.411 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.417+0000 7f3b1818d640 10 monclient: init 2024-03-29T11:28:42.411 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.418+0000 7f3b1818d640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:42.411 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.418+0000 7f3b1818d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:42.411 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.418+0000 7f3b1818d640 10 monclient(hunting): picked mon.noname-c con 0x7f3b100a9d70 addr v1:172.21.15.135:6790/0 2024-03-29T11:28:42.411 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.418+0000 7f3b15f29640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f3b100a9d70 0x7f3b1013b110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:35056/0 (socket says 172.21.15.135:35056) 2024-03-29T11:28:42.411 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.418+0000 7f3b15f29640 1 -- 172.21.15.135:0/1909960090 learned_addr learned my addr 172.21.15.135:0/1909960090 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:28:42.411 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.418+0000 7f3b067fc640 10 client.?.objecter ms_handle_connect 0x7f3b100a9d70 2024-03-29T11:28:42.411 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.418+0000 7f3b067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:42.421 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.418+0000 7f3b1818d640 10 monclient(hunting): picked mon.noname-b con 0x7f3b1013f650 addr v1:172.21.15.194:6789/0 2024-03-29T11:28:42.421 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.419+0000 7f3b1818d640 10 monclient(hunting): picked mon.noname-a con 0x7f3b101457f0 addr v1:172.21.15.135:6789/0 2024-03-29T11:28:42.421 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.419+0000 7f3b1818d640 1 -- 172.21.15.135:0/1909960090 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b1013b970 con 0x7f3b101457f0 2024-03-29T11:28:42.422 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.419+0000 7f3b1818d640 1 -- 172.21.15.135:0/1909960090 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b10139af0 con 0x7f3b1013f650 2024-03-29T11:28:42.422 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.419+0000 7f3b1818d640 1 -- 172.21.15.135:0/1909960090 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b10139c80 con 0x7f3b100a9d70 2024-03-29T11:28:42.422 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.419+0000 7f3b1818d640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:42.422 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.419+0000 7f3b1818d640 10 monclient(hunting): authenticate will time out at 1029.787859s 2024-03-29T11:28:42.422 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.423+0000 7f3b067fc640 1 -- 172.21.15.135:0/1909960090 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 408280512 0 0) 0x7f3b10139c80 con 0x7f3b100a9d70 2024-03-29T11:28:42.422 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.423+0000 7f3b067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:42.422 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.423+0000 7f3b067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:42.422 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.423+0000 7f3b067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:42.422 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.423+0000 7f3b067fc640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:28:42.422 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.423+0000 7f3b067fc640 1 -- 172.21.15.135:0/1909960090 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3af4004eb0 con 0x7f3b100a9d70 2024-03-29T11:28:42.422 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.423+0000 7f3b067fc640 10 client.?.objecter ms_handle_connect 0x7f3b101457f0 2024-03-29T11:28:42.423 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.423+0000 7f3b067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:42.423 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.424+0000 7f3b067fc640 1 -- 172.21.15.135:0/1909960090 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496825373 0 0) 0x7f3b1013b970 con 0x7f3b101457f0 2024-03-29T11:28:42.423 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.424+0000 7f3b067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:42.423 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.424+0000 7f3b067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:42.423 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.424+0000 7f3b067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:42.423 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.424+0000 7f3b067fc640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:28:42.423 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.424+0000 7f3b067fc640 1 -- 172.21.15.135:0/1909960090 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b10139c80 con 0x7f3b101457f0 2024-03-29T11:28:42.423 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.424+0000 7f3b067fc640 1 -- 172.21.15.135:0/1909960090 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2949979045 0 0) 0x7f3b10139c80 con 0x7f3b101457f0 2024-03-29T11:28:42.423 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.424+0000 7f3b067fc640 1 -- 172.21.15.135:0/1909960090 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b1013b970 con 0x7f3b101457f0 2024-03-29T11:28:42.423 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.424+0000 7f3b067fc640 1 -- 172.21.15.135:0/1909960090 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f3b00001510 con 0x7f3b101457f0 2024-03-29T11:28:42.423 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.424+0000 7f3b067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:42.423 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.424+0000 7f3b067fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:42.423 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.424+0000 7f3b067fc640 10 monclient(hunting): dump: 2024-03-29T11:28:42.423 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:42.424 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:42.424 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:42.424 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:42.424 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:42.424 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:42.424 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:42.424 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:42.424 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:42.424 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:42.425 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.425+0000 7f3b067fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:42.425 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.425+0000 7f3b067fc640 1 -- 172.21.15.135:0/1909960090 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 460674076 0 0) 0x7f3b1013b970 con 0x7f3b101457f0 2024-03-29T11:28:42.425 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.425+0000 7f3b067fc640 1 -- 172.21.15.135:0/1909960090 >> v1:172.21.15.135:6790/0 conn(0x7f3b100a9d70 legacy=0x7f3b1013b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:42.425 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.425+0000 7f3b067fc640 1 -- 172.21.15.135:0/1909960090 >> v1:172.21.15.194:6789/0 conn(0x7f3b1013f650 legacy=0x7f3b101393e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:42.425 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.425+0000 7f3b067fc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:42.425 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.425+0000 7f3b067fc640 1 monclient: found mon.a 2024-03-29T11:28:42.425 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.425+0000 7f3b067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:42.425 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.425+0000 7f3b067fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:28:42.425 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.425+0000 7f3b067fc640 1 -- 172.21.15.135:0/1909960090 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b10138690 con 0x7f3b101457f0 2024-03-29T11:28:42.425 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.425+0000 7f3b067fc640 10 monclient: _finish_auth 0 2024-03-29T11:28:42.425 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.425+0000 7f3b067fc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:42.425 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.425+0000 7f3b067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:42.425 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.425+0000 7f3b1818d640 5 monclient: authenticate success, global_id 4392 2024-03-29T11:28:42.426 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.425+0000 7f3b1818d640 10 monclient: _renew_subs 2024-03-29T11:28:42.426 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.425+0000 7f3b1818d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:28:42.426 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.425+0000 7f3b1818d640 1 -- 172.21.15.135:0/1909960090 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b10138940 con 0x7f3b101457f0 2024-03-29T11:28:42.426 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.425+0000 7f3b1818d640 10 client.4392.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:42.426 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.425+0000 7f3b1818d640 10 monclient: _renew_subs 2024-03-29T11:28:42.426 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.425+0000 7f3b1818d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:28:42.426 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.425+0000 7f3b1818d640 1 -- 172.21.15.135:0/1909960090 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b10142540 con 0x7f3b101457f0 2024-03-29T11:28:42.426 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.425+0000 7f3b1818d640 1 librados: init done 2024-03-29T11:28:42.426 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.426+0000 7f3b067fc640 1 -- 172.21.15.135:0/1909960090 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b00001670 con 0x7f3b101457f0 2024-03-29T11:28:42.426 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.426+0000 7f3b067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:42.426 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.426+0000 7f3b067fc640 1 -- 172.21.15.135:0/1909960090 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f3b00001cc0 con 0x7f3b101457f0 2024-03-29T11:28:42.426 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.426+0000 7f3b067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:42.426 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.426+0000 7f3b067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:42.426 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.426+0000 7f3b067fc640 10 monclient: dump: 2024-03-29T11:28:42.427 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:42.427 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:42.427 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:42.427 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:42.427 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:42.427 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:42.427 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:42.427 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:42.427 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:42.427 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:42.427 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.427+0000 7f3b07fff640 4 set_mon_vals no callback set 2024-03-29T11:28:42.427 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.429+0000 7f3b067fc640 1 -- 172.21.15.135:0/1909960090 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f3b00019700 con 0x7f3b101457f0 2024-03-29T11:28:42.427 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.429+0000 7f3b067fc640 1 -- 172.21.15.135:0/1909960090 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f3b00083730 con 0x7f3b101457f0 2024-03-29T11:28:42.427 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.429+0000 7f3b067fc640 10 client.4392.objecter ms_dispatch 0x7f3b10000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:42.428 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.430+0000 7f3b067fc640 3 client.4392.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:42.428 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.430+0000 7f3b067fc640 3 client.4392.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:42.428 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.430+0000 7f3b15f29640 10 monclient: get_auth_request con 0x7f3af406dd20 auth_method 0 2024-03-29T11:28:42.428 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.430+0000 7f3b067fc640 20 client.4392.objecter dump_active .. 0 homeless 2024-03-29T11:28:42.428 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.430+0000 7f3b1818d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:42.428 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.430+0000 7f3b1818d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:42.428 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.430+0000 7f3b067fc640 10 client.4392.objecter ms_handle_connect 0x7f3af406dd20 2024-03-29T11:28:42.428 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.430+0000 7f3b1818d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:28:42.428 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.430+0000 7f3b1818d640 1 -- 172.21.15.135:0/1909960090 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af8005080 con 0x7f3b101457f0 2024-03-29T11:28:42.429 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.439+0000 7f3b067fc640 1 -- 172.21.15.135:0/1909960090 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3b0004ee40 con 0x7f3b101457f0 2024-03-29T11:28:42.429 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.439+0000 7f3b067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:42.429 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.439+0000 7f3b067fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:42.759 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.769+0000 7f3b1818d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:42.760 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.769+0000 7f3b1818d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:42.760 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.769+0000 7f3b1818d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:28:42.760 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.769+0000 7f3b1818d640 1 -- 172.21.15.135:0/1909960090 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3af8005330 con 0x7f3b101457f0 2024-03-29T11:28:42.760 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.771+0000 7f3b067fc640 1 -- 172.21.15.135:0/1909960090 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f3b00050f40 con 0x7f3b101457f0 2024-03-29T11:28:42.760 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.771+0000 7f3b067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:42.760 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.771+0000 7f3b067fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:42.762 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:28:42.763 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:42.766 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.776+0000 7f3b1818d640 10 librados: watch_flush enter 2024-03-29T11:28:42.766 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.777+0000 7f3b1818d640 10 librados: watch_flush exit 2024-03-29T11:28:42.766 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.777+0000 7f3b1818d640 20 client.4392.objecter shutdown clearing up homeless session... 2024-03-29T11:28:42.766 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.777+0000 7f3b1818d640 10 client.4392.objecter successfully canceled tick 2024-03-29T11:28:42.768 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.778+0000 7f3b1818d640 1 -- 172.21.15.135:0/1909960090 >> v1:172.21.15.135:6816/3469875285 conn(0x7f3af406dd20 legacy=0x7f3af40701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:42.768 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.778+0000 7f3b1818d640 10 monclient: shutdown 2024-03-29T11:28:42.768 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.778+0000 7f3b1818d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:42.768 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.778+0000 7f3b1818d640 1 -- 172.21.15.135:0/1909960090 >> v1:172.21.15.135:6789/0 conn(0x7f3b101457f0 legacy=0x7f3b100b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:42.768 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.778+0000 7f3b1818d640 1 -- 172.21.15.135:0/1909960090 shutdown_connections 2024-03-29T11:28:42.768 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.778+0000 7f3b1818d640 1 -- 172.21.15.135:0/1909960090 >> 172.21.15.135:0/1909960090 conn(0x7f3b10066a00 msgr2=0x7f3b100b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:42.769 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.779+0000 7f3b1818d640 1 -- 172.21.15.135:0/1909960090 shutdown_connections 2024-03-29T11:28:42.769 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.779+0000 7f3b1818d640 1 -- 172.21.15.135:0/1909960090 wait complete. 2024-03-29T11:28:42.769 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:42.779+0000 7f3b1818d640 1 librados: shutdown 2024-03-29T11:28:42.784 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:49.590 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:49.743 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.751+0000 7f4dc7cf4640 10 monclient: get_monmap_and_config 2024-03-29T11:28:49.743 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.751+0000 7f4dc7cf4640 10 monclient: build_initial_monmap 2024-03-29T11:28:49.743 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.751+0000 7f4dc7cf4640 10 monclient: monmap: 2024-03-29T11:28:49.743 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:28:49.744 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:49.744 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:28:49.752244+0000 2024-03-29T11:28:49.744 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:28:49.752244+0000 2024-03-29T11:28:49.744 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:49.744 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:49.744 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:28:49.744 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:28:49.744 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:28:49.744 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:49.744 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.753+0000 7f4dc7cf4640 1 Processor -- start 2024-03-29T11:28:49.744 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.753+0000 7f4dc7cf4640 1 -- start start 2024-03-29T11:28:49.744 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.753+0000 7f4dc7cf4640 10 monclient: init 2024-03-29T11:28:49.748 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.754+0000 7f4dc7cf4640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:49.748 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.754+0000 7f4dc7cf4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:49.748 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.754+0000 7f4dc7cf4640 10 monclient(hunting): picked mon.noname-c con 0x7f4dc00a9750 addr v1:172.21.15.194:6789/0 2024-03-29T11:28:49.748 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.754+0000 7f4dc7cf4640 10 monclient(hunting): picked mon.noname-b con 0x7f4dc0149ac0 addr v1:172.21.15.135:6790/0 2024-03-29T11:28:49.748 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.754+0000 7f4dc7cf4640 10 monclient(hunting): picked mon.noname-a con 0x7f4dc014fe50 addr v1:172.21.15.135:6789/0 2024-03-29T11:28:49.748 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.754+0000 7f4dc7cf4640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc00aa4a0 con 0x7f4dc014fe50 2024-03-29T11:28:49.748 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.754+0000 7f4dc7cf4640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc014a680 con 0x7f4dc00a9750 2024-03-29T11:28:49.749 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.754+0000 7f4dc7cf4640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc00af290 con 0x7f4dc0149ac0 2024-03-29T11:28:49.749 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.754+0000 7f4dc7cf4640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:49.749 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.754+0000 7f4dc7cf4640 10 monclient(hunting): authenticate will time out at 1037.122790s 2024-03-29T11:28:49.749 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.758+0000 7f4dc528f640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f4dc0149ac0 0x7f4dc0149f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:39187/0 (socket says 172.21.15.135:39187) 2024-03-29T11:28:49.750 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.758+0000 7f4dc528f640 1 -- 172.21.15.135:0/4162170169 learned_addr learned my addr 172.21.15.135:0/4162170169 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:28:49.750 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.759+0000 7f4dc4a8e640 1 -- 172.21.15.135:0/4162170169 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110697185 0 0) 0x7f4dc00af290 con 0x7f4dc0149ac0 2024-03-29T11:28:49.750 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.759+0000 7f4dc4a8e640 10 monclient(hunting): _negotiate 2024-03-29T11:28:49.750 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.759+0000 7f4dc4a8e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:49.750 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.759+0000 7f4dc4a8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:49.750 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.759+0000 7f4dc4a8e640 10 monclient(hunting): my global_id is 4388 2024-03-29T11:28:49.750 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.759+0000 7f4dc4a8e640 1 -- 172.21.15.135:0/4162170169 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dac003520 con 0x7f4dc0149ac0 2024-03-29T11:28:49.751 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.759+0000 7f4dc4a8e640 1 -- 172.21.15.135:0/4162170169 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3507012060 0 0) 0x7f4dac003520 con 0x7f4dc0149ac0 2024-03-29T11:28:49.751 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.759+0000 7f4dc4a8e640 1 -- 172.21.15.135:0/4162170169 >> v1:172.21.15.194:6789/0 conn(0x7f4dc00a9750 legacy=0x7f4dc01436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:49.751 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.759+0000 7f4dc4a8e640 1 -- 172.21.15.135:0/4162170169 >> v1:172.21.15.135:6789/0 conn(0x7f4dc014fe50 legacy=0x7f4dc0152240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:49.753 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.761+0000 7f4dc4a8e640 10 monclient: _finish_hunting 0 2024-03-29T11:28:49.753 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.761+0000 7f4dc4a8e640 1 monclient: found mon.noname-b 2024-03-29T11:28:49.753 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.762+0000 7f4dc4a8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:49.753 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.762+0000 7f4dc4a8e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:28:49.753 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.762+0000 7f4dc4a8e640 1 -- 172.21.15.135:0/4162170169 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc00afb40 con 0x7f4dc0149ac0 2024-03-29T11:28:49.753 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.762+0000 7f4dc4a8e640 10 monclient: _finish_auth 0 2024-03-29T11:28:49.753 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.762+0000 7f4dc4a8e640 10 monclient: _check_auth_tickets 2024-03-29T11:28:49.753 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.762+0000 7f4dc4a8e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:49.753 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.762+0000 7f4dc7cf4640 5 monclient: authenticate success, global_id 4388 2024-03-29T11:28:49.753 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.762+0000 7f4dc7cf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:49.753 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.762+0000 7f4dc4a8e640 1 -- 172.21.15.135:0/4162170169 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f4db8001a10 con 0x7f4dc0149ac0 2024-03-29T11:28:49.754 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.762+0000 7f4dc4a8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:49.754 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.762+0000 7f4dc4a8e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:49.754 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.762+0000 7f4dc4a8e640 10 monclient: dump: 2024-03-29T11:28:49.754 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:49.754 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:49.754 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:49.754 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:49.754 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:49.754 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:49.754 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:49.754 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:49.754 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:49.754 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:49.754 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.762+0000 7f4dc7cf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:49.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.762+0000 7f4dc4a8e640 1 -- 172.21.15.135:0/4162170169 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4db8001ca0 con 0x7f4dc0149ac0 2024-03-29T11:28:49.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.762+0000 7f4dc4a8e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:49.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.762+0000 7f4dc7cf4640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:49.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.762+0000 7f4dc7cf4640 4 set_mon_vals no callback set 2024-03-29T11:28:49.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.763+0000 7f4dc4a8e640 1 -- 172.21.15.135:0/4162170169 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f4db8002030 con 0x7f4dc0149ac0 2024-03-29T11:28:49.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.763+0000 7f4dc4a8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:49.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.763+0000 7f4dc4a8e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:49.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.763+0000 7f4dc4a8e640 10 monclient: dump: 2024-03-29T11:28:49.755 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:49.755 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:49.755 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:49.755 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:49.755 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:49.755 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:49.756 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:49.756 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:49.756 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:49.756 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:49.756 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.763+0000 7f4dc7cf4640 10 monclient: shutdown 2024-03-29T11:28:49.756 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.763+0000 7f4dc7cf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:49.756 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.763+0000 7f4dc7cf4640 1 -- 172.21.15.135:0/4162170169 >> v1:172.21.15.135:6790/0 conn(0x7f4dc0149ac0 legacy=0x7f4dc0149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:49.756 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.763+0000 7f4dc7cf4640 1 -- 172.21.15.135:0/4162170169 shutdown_connections 2024-03-29T11:28:49.756 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.763+0000 7f4dc7cf4640 1 -- 172.21.15.135:0/4162170169 >> 172.21.15.135:0/4162170169 conn(0x7f4dc00b35e0 msgr2=0x7f4dc00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:49.757 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.763+0000 7f4dc7cf4640 1 -- 172.21.15.135:0/4162170169 shutdown_connections 2024-03-29T11:28:49.759 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.763+0000 7f4dc7cf4640 1 -- 172.21.15.135:0/4162170169 wait complete. 2024-03-29T11:28:49.759 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.764+0000 7f4dc7cf4640 10 monclient: build_initial_monmap 2024-03-29T11:28:49.759 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.764+0000 7f4dc7cf4640 10 monclient: monmap: 2024-03-29T11:28:49.759 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:28:49.760 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:49.760 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:28:49.760 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:28:49.760 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:49.760 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:49.760 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:28:49.760 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:28:49.760 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:28:49.760 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:49.760 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.764+0000 7f4dc7cf4640 1 librados: starting msgr at 2024-03-29T11:28:49.760 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.764+0000 7f4dc7cf4640 1 librados: starting objecter 2024-03-29T11:28:49.760 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.764+0000 7f4dc7cf4640 1 Processor -- start 2024-03-29T11:28:49.760 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.765+0000 7f4dc7cf4640 1 -- start start 2024-03-29T11:28:49.761 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.765+0000 7f4dc7cf4640 1 librados: setting wanted keys 2024-03-29T11:28:49.761 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.765+0000 7f4dc7cf4640 1 librados: calling monclient init 2024-03-29T11:28:49.761 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.765+0000 7f4dc7cf4640 10 monclient: init 2024-03-29T11:28:49.761 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.765+0000 7f4dc7cf4640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:49.761 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.765+0000 7f4dc7cf4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:49.761 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.765+0000 7f4dc7cf4640 10 monclient(hunting): picked mon.noname-a con 0x7f4dc00a9750 addr v1:172.21.15.135:6789/0 2024-03-29T11:28:49.761 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.765+0000 7f4dc7cf4640 10 monclient(hunting): picked mon.noname-c con 0x7f4dc0149ac0 addr v1:172.21.15.135:6790/0 2024-03-29T11:28:49.761 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.765+0000 7f4dc7cf4640 10 monclient(hunting): picked mon.noname-b con 0x7f4dc014fe50 addr v1:172.21.15.194:6789/0 2024-03-29T11:28:49.761 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.765+0000 7f4dc7cf4640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc00ab7c0 con 0x7f4dc00a9750 2024-03-29T11:28:49.761 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.765+0000 7f4dc7cf4640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc0144fa0 con 0x7f4dc014fe50 2024-03-29T11:28:49.761 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.765+0000 7f4dc7cf4640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dc013be10 con 0x7f4dc0149ac0 2024-03-29T11:28:49.761 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.765+0000 7f4dc7cf4640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:49.762 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.765+0000 7f4dc7cf4640 10 monclient(hunting): authenticate will time out at 1037.133991s 2024-03-29T11:28:49.762 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.767+0000 7f4dc6291640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f4dc014fe50 0x7f4dc01df180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:35754/0 (socket says 172.21.15.135:35754) 2024-03-29T11:28:49.762 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.767+0000 7f4dc6291640 1 -- 172.21.15.135:0/1048220587 learned_addr learned my addr 172.21.15.135:0/1048220587 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:28:49.762 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.767+0000 7f4db67fc640 10 client.?.objecter ms_handle_connect 0x7f4dc014fe50 2024-03-29T11:28:49.762 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.767+0000 7f4db67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:49.762 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.767+0000 7f4db67fc640 1 -- 172.21.15.135:0/1048220587 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4175615956 0 0) 0x7f4dc0144fa0 con 0x7f4dc014fe50 2024-03-29T11:28:49.762 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.767+0000 7f4db67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:49.762 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.767+0000 7f4db67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:49.762 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.767+0000 7f4db67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:49.762 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.767+0000 7f4db67fc640 10 monclient(hunting): my global_id is 4384 2024-03-29T11:28:49.762 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.767+0000 7f4db67fc640 1 -- 172.21.15.135:0/1048220587 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4da4004eb0 con 0x7f4dc014fe50 2024-03-29T11:28:49.762 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.768+0000 7f4db67fc640 1 -- 172.21.15.135:0/1048220587 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3677386614 0 0) 0x7f4da4004eb0 con 0x7f4dc014fe50 2024-03-29T11:28:49.763 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.768+0000 7f4db67fc640 1 -- 172.21.15.135:0/1048220587 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4dc0144fa0 con 0x7f4dc014fe50 2024-03-29T11:28:49.763 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.769+0000 7f4db67fc640 1 -- 172.21.15.135:0/1048220587 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f4dbc001750 con 0x7f4dc014fe50 2024-03-29T11:28:49.763 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.769+0000 7f4db67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:49.763 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.769+0000 7f4db67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:49.763 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.769+0000 7f4db67fc640 10 monclient(hunting): dump: 2024-03-29T11:28:49.763 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:49.763 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:49.763 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:49.763 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:49.763 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:49.763 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:49.763 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:49.763 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:49.764 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:49.764 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:49.764 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.769+0000 7f4db67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:49.764 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.769+0000 7f4db67fc640 1 -- 172.21.15.135:0/1048220587 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1753693250 0 0) 0x7f4dc0144fa0 con 0x7f4dc014fe50 2024-03-29T11:28:49.764 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.769+0000 7f4db67fc640 1 -- 172.21.15.135:0/1048220587 >> v1:172.21.15.135:6790/0 conn(0x7f4dc0149ac0 legacy=0x7f4dc01469c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:49.764 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.769+0000 7f4db67fc640 1 -- 172.21.15.135:0/1048220587 >> v1:172.21.15.135:6789/0 conn(0x7f4dc00a9750 legacy=0x7f4dc013cb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:49.768 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.770+0000 7f4db67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:49.768 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.770+0000 7f4db67fc640 1 monclient: found mon.b 2024-03-29T11:28:49.768 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.770+0000 7f4db67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:49.768 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.770+0000 7f4db67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:28:49.768 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.770+0000 7f4db67fc640 1 -- 172.21.15.135:0/1048220587 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc0144340 con 0x7f4dc014fe50 2024-03-29T11:28:49.768 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.770+0000 7f4db67fc640 10 monclient: _finish_auth 0 2024-03-29T11:28:49.768 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.770+0000 7f4db67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:49.768 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.770+0000 7f4db67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:49.768 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.770+0000 7f4dc7cf4640 5 monclient: authenticate success, global_id 4384 2024-03-29T11:28:49.768 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.770+0000 7f4dc7cf4640 10 monclient: _renew_subs 2024-03-29T11:28:49.768 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.770+0000 7f4dc7cf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:28:49.769 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.770+0000 7f4dc7cf4640 1 -- 172.21.15.135:0/1048220587 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc013d230 con 0x7f4dc014fe50 2024-03-29T11:28:49.769 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.770+0000 7f4dc7cf4640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:49.769 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.770+0000 7f4dc7cf4640 10 monclient: _renew_subs 2024-03-29T11:28:49.769 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.770+0000 7f4dc7cf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:28:49.769 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.770+0000 7f4dc7cf4640 1 -- 172.21.15.135:0/1048220587 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc01472b0 con 0x7f4dc014fe50 2024-03-29T11:28:49.769 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.770+0000 7f4dc7cf4640 1 librados: init done 2024-03-29T11:28:49.769 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.771+0000 7f4db67fc640 1 -- 172.21.15.135:0/1048220587 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4dbc0018b0 con 0x7f4dc014fe50 2024-03-29T11:28:49.769 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.771+0000 7f4db67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:49.769 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.771+0000 7f4db7fff640 4 set_mon_vals no callback set 2024-03-29T11:28:49.769 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.771+0000 7f4db67fc640 1 -- 172.21.15.135:0/1048220587 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f4dbc004a00 con 0x7f4dc014fe50 2024-03-29T11:28:49.769 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.771+0000 7f4db67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:49.769 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.771+0000 7f4db67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:49.769 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.771+0000 7f4db67fc640 10 monclient: dump: 2024-03-29T11:28:49.770 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:49.770 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:49.770 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:49.770 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:49.770 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:49.770 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:49.770 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:49.770 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:49.771 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:49.771 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:49.771 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.773+0000 7f4dc7cf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:49.771 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.773+0000 7f4dc7cf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:49.771 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.773+0000 7f4dc7cf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:28:49.771 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.773+0000 7f4dc7cf4640 1 -- 172.21.15.135:0/1048220587 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4da8005080 con 0x7f4dc014fe50 2024-03-29T11:28:49.771 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.774+0000 7f4db67fc640 1 -- 172.21.15.135:0/1048220587 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f4dbc004b60 con 0x7f4dc014fe50 2024-03-29T11:28:49.771 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.774+0000 7f4db67fc640 1 -- 172.21.15.135:0/1048220587 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f4dbc084a20 con 0x7f4dc014fe50 2024-03-29T11:28:49.771 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.775+0000 7f4db67fc640 10 client.4384.objecter ms_dispatch 0x7f4dc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:49.771 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.775+0000 7f4db67fc640 3 client.4384.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:49.771 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.775+0000 7f4db67fc640 3 client.4384.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:49.771 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.775+0000 7f4db67fc640 20 client.4384.objecter dump_active .. 0 homeless 2024-03-29T11:28:49.772 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.776+0000 7f4dc5a90640 10 monclient: get_auth_request con 0x7f4da406dc20 auth_method 0 2024-03-29T11:28:49.772 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.777+0000 7f4db67fc640 10 client.4384.objecter ms_handle_connect 0x7f4da406dc20 2024-03-29T11:28:49.772 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.781+0000 7f4db67fc640 1 -- 172.21.15.135:0/1048220587 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4dbc050130 con 0x7f4dc014fe50 2024-03-29T11:28:49.772 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.782+0000 7f4db67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:49.772 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:49.782+0000 7f4db67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:50.099 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:50.108+0000 7f4dc7cf4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:50.099 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:50.108+0000 7f4dc7cf4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:50.100 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:50.108+0000 7f4dc7cf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:28:50.100 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:50.108+0000 7f4dc7cf4640 1 -- 172.21.15.135:0/1048220587 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4da8005330 con 0x7f4dc014fe50 2024-03-29T11:28:50.101 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:50.110+0000 7f4db67fc640 1 -- 172.21.15.135:0/1048220587 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f4dbc052230 con 0x7f4dc014fe50 2024-03-29T11:28:50.101 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:50.110+0000 7f4db67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:50.101 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:50.110+0000 7f4db67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:50.103 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:28:50.103 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:50.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:50.113+0000 7f4dc7cf4640 10 librados: watch_flush enter 2024-03-29T11:28:50.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:50.113+0000 7f4dc7cf4640 10 librados: watch_flush exit 2024-03-29T11:28:50.105 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:50.113+0000 7f4dc7cf4640 20 client.4384.objecter shutdown clearing up homeless session... 2024-03-29T11:28:50.105 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:50.113+0000 7f4dc7cf4640 10 client.4384.objecter successfully canceled tick 2024-03-29T11:28:50.105 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:50.114+0000 7f4dc7cf4640 1 -- 172.21.15.135:0/1048220587 >> v1:172.21.15.135:6816/3469875285 conn(0x7f4da406dc20 legacy=0x7f4da40700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:50.105 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:50.114+0000 7f4dc7cf4640 10 monclient: shutdown 2024-03-29T11:28:50.105 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:50.114+0000 7f4dc7cf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:50.105 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:50.114+0000 7f4dc7cf4640 1 -- 172.21.15.135:0/1048220587 >> v1:172.21.15.194:6789/0 conn(0x7f4dc014fe50 legacy=0x7f4dc01df180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:50.105 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:50.114+0000 7f4dc7cf4640 1 -- 172.21.15.135:0/1048220587 shutdown_connections 2024-03-29T11:28:50.105 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:50.114+0000 7f4dc7cf4640 1 -- 172.21.15.135:0/1048220587 >> 172.21.15.135:0/1048220587 conn(0x7f4dc00b35e0 msgr2=0x7f4dc00b2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:50.106 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:50.114+0000 7f4dc7cf4640 1 -- 172.21.15.135:0/1048220587 shutdown_connections 2024-03-29T11:28:50.106 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:50.115+0000 7f4dc7cf4640 1 -- 172.21.15.135:0/1048220587 wait complete. 2024-03-29T11:28:50.106 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:50.115+0000 7f4dc7cf4640 1 librados: shutdown 2024-03-29T11:28:50.124 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:56.931 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:57.064 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.073+0000 7f1b40b32640 10 monclient: get_monmap_and_config 2024-03-29T11:28:57.064 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.073+0000 7f1b40b32640 10 monclient: build_initial_monmap 2024-03-29T11:28:57.065 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.073+0000 7f1b40b32640 10 monclient: monmap: 2024-03-29T11:28:57.065 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:28:57.065 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:57.065 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:28:57.074988+0000 2024-03-29T11:28:57.065 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:28:57.074988+0000 2024-03-29T11:28:57.065 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:57.065 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:57.065 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:28:57.065 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:28:57.065 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:28:57.065 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:57.066 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.074+0000 7f1b40b32640 1 Processor -- start 2024-03-29T11:28:57.066 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.074+0000 7f1b40b32640 1 -- start start 2024-03-29T11:28:57.066 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.074+0000 7f1b40b32640 10 monclient: init 2024-03-29T11:28:57.067 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.075+0000 7f1b40b32640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:57.067 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.075+0000 7f1b40b32640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:57.067 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.075+0000 7f1b40b32640 10 monclient(hunting): picked mon.noname-c con 0x7f1b3c0b0860 addr v1:172.21.15.194:6789/0 2024-03-29T11:28:57.067 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.076+0000 7f1b40b32640 10 monclient(hunting): picked mon.noname-a con 0x7f1b3c072450 addr v1:172.21.15.135:6789/0 2024-03-29T11:28:57.067 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.076+0000 7f1b3a59c640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f1b3c0b0860 0x7f1b3c0acfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:52718/0 (socket says 172.21.15.135:52718) 2024-03-29T11:28:57.067 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.076+0000 7f1b3a59c640 1 -- 172.21.15.135:0/507428306 learned_addr learned my addr 172.21.15.135:0/507428306 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:28:57.067 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.076+0000 7f1b40b32640 10 monclient(hunting): picked mon.noname-b con 0x7f1b3c13ee40 addr v1:172.21.15.135:6790/0 2024-03-29T11:28:57.067 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.076+0000 7f1b40b32640 1 -- 172.21.15.135:0/507428306 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3c0ad840 con 0x7f1b3c072450 2024-03-29T11:28:57.067 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.076+0000 7f1b40b32640 1 -- 172.21.15.135:0/507428306 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3c13c830 con 0x7f1b3c0b0860 2024-03-29T11:28:57.067 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.076+0000 7f1b40b32640 1 -- 172.21.15.135:0/507428306 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3c13c9c0 con 0x7f1b3c13ee40 2024-03-29T11:28:57.067 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.076+0000 7f1b40b32640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:57.068 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.076+0000 7f1b40b32640 10 monclient(hunting): authenticate will time out at 1044.445410s 2024-03-29T11:28:57.068 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.076+0000 7f1b3959a640 1 -- 172.21.15.135:0/507428306 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688545611 0 0) 0x7f1b3c0ad840 con 0x7f1b3c072450 2024-03-29T11:28:57.068 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.076+0000 7f1b3959a640 10 monclient(hunting): _negotiate 2024-03-29T11:28:57.068 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.076+0000 7f1b3959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:57.068 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.076+0000 7f1b3959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:57.068 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.076+0000 7f1b3959a640 10 monclient(hunting): my global_id is 4395 2024-03-29T11:28:57.068 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.076+0000 7f1b3959a640 1 -- 172.21.15.135:0/507428306 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b24005730 con 0x7f1b3c072450 2024-03-29T11:28:57.069 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.077+0000 7f1b3959a640 1 -- 172.21.15.135:0/507428306 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3875666103 0 0) 0x7f1b24005730 con 0x7f1b3c072450 2024-03-29T11:28:57.069 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.077+0000 7f1b3959a640 1 -- 172.21.15.135:0/507428306 >> v1:172.21.15.135:6790/0 conn(0x7f1b3c13ee40 legacy=0x7f1b3c13f260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:57.070 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.079+0000 7f1b3959a640 1 -- 172.21.15.135:0/507428306 >> v1:172.21.15.194:6789/0 conn(0x7f1b3c0b0860 legacy=0x7f1b3c0acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.070 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.079+0000 7f1b3959a640 10 monclient: _finish_hunting 0 2024-03-29T11:28:57.070 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.079+0000 7f1b3959a640 1 monclient: found mon.noname-a 2024-03-29T11:28:57.070 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.079+0000 7f1b3959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:57.070 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.079+0000 7f1b3959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.135:6789/0 2024-03-29T11:28:57.070 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.079+0000 7f1b3959a640 1 -- 172.21.15.135:0/507428306 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3c13f970 con 0x7f1b3c072450 2024-03-29T11:28:57.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.079+0000 7f1b3959a640 10 monclient: _finish_auth 0 2024-03-29T11:28:57.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.079+0000 7f1b3959a640 10 monclient: _check_auth_tickets 2024-03-29T11:28:57.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.079+0000 7f1b3959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:57.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.079+0000 7f1b3959a640 1 -- 172.21.15.135:0/507428306 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f1b2c001690 con 0x7f1b3c072450 2024-03-29T11:28:57.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.079+0000 7f1b3959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:57.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.079+0000 7f1b3959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:57.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.079+0000 7f1b3959a640 10 monclient: dump: 2024-03-29T11:28:57.071 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:57.071 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:57.071 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:57.071 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:57.071 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:57.071 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:57.071 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:57.072 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:57.072 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:57.072 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:57.072 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.079+0000 7f1b40b32640 5 monclient: authenticate success, global_id 4395 2024-03-29T11:28:57.072 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.079+0000 7f1b40b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:57.072 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.079+0000 7f1b3959a640 1 -- 172.21.15.135:0/507428306 <== mon.0 v1:172.21.15.135:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b2c001860 con 0x7f1b3c072450 2024-03-29T11:28:57.072 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.079+0000 7f1b3959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:57.072 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.079+0000 7f1b3959a640 1 -- 172.21.15.135:0/507428306 <== mon.0 v1:172.21.15.135:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f1b2c001cb0 con 0x7f1b3c072450 2024-03-29T11:28:57.072 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.079+0000 7f1b3959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:57.072 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.079+0000 7f1b3959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:57.072 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.079+0000 7f1b3959a640 10 monclient: dump: 2024-03-29T11:28:57.072 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:57.072 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:57.072 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:57.073 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:57.073 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:57.073 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:57.073 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:57.073 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:57.073 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:57.073 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:57.073 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.079+0000 7f1b40b32640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:57.073 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.079+0000 7f1b40b32640 4 set_mon_vals no callback set 2024-03-29T11:28:57.077 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.080+0000 7f1b40b32640 10 monclient: shutdown 2024-03-29T11:28:57.077 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.080+0000 7f1b40b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:57.077 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.080+0000 7f1b40b32640 1 -- 172.21.15.135:0/507428306 >> v1:172.21.15.135:6789/0 conn(0x7f1b3c072450 legacy=0x7f1b3c13c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.077 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.080+0000 7f1b40b32640 1 -- 172.21.15.135:0/507428306 shutdown_connections 2024-03-29T11:28:57.077 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.081+0000 7f1b40b32640 1 -- 172.21.15.135:0/507428306 >> 172.21.15.135:0/507428306 conn(0x7f1b3c06b660 msgr2=0x7f1b3c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:57.077 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.081+0000 7f1b40b32640 1 -- 172.21.15.135:0/507428306 shutdown_connections 2024-03-29T11:28:57.077 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.081+0000 7f1b40b32640 1 -- 172.21.15.135:0/507428306 wait complete. 2024-03-29T11:28:57.077 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.082+0000 7f1b40b32640 10 monclient: build_initial_monmap 2024-03-29T11:28:57.077 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.082+0000 7f1b40b32640 10 monclient: monmap: 2024-03-29T11:28:57.077 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:28:57.077 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:57.078 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:28:57.078 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:28:57.078 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:57.078 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:57.078 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:28:57.078 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:28:57.078 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:28:57.078 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:57.078 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.082+0000 7f1b40b32640 1 librados: starting msgr at 2024-03-29T11:28:57.078 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.082+0000 7f1b40b32640 1 librados: starting objecter 2024-03-29T11:28:57.078 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.082+0000 7f1b40b32640 1 Processor -- start 2024-03-29T11:28:57.078 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.082+0000 7f1b40b32640 1 -- start start 2024-03-29T11:28:57.078 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.082+0000 7f1b40b32640 1 librados: setting wanted keys 2024-03-29T11:28:57.078 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.082+0000 7f1b40b32640 1 librados: calling monclient init 2024-03-29T11:28:57.079 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.082+0000 7f1b40b32640 10 monclient: init 2024-03-29T11:28:57.079 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.083+0000 7f1b40b32640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:57.079 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.083+0000 7f1b40b32640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:57.079 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.083+0000 7f1b40b32640 10 monclient(hunting): picked mon.noname-c con 0x7f1b3c072450 addr v1:172.21.15.135:6790/0 2024-03-29T11:28:57.079 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.083+0000 7f1b40b32640 10 monclient(hunting): picked mon.noname-a con 0x7f1b3c0b0860 addr v1:172.21.15.135:6789/0 2024-03-29T11:28:57.079 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.083+0000 7f1b39d9b640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f1b3c0b0860 0x7f1b3c1d2e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:47918/0 (socket says 172.21.15.135:47918) 2024-03-29T11:28:57.079 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.083+0000 7f1b39d9b640 1 -- 172.21.15.135:0/2961796460 learned_addr learned my addr 172.21.15.135:0/2961796460 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:28:57.079 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.083+0000 7f1b2affd640 10 client.?.objecter ms_handle_connect 0x7f1b3c0b0860 2024-03-29T11:28:57.079 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b40b32640 10 monclient(hunting): picked mon.noname-b con 0x7f1b3c13ee40 addr v1:172.21.15.194:6789/0 2024-03-29T11:28:57.079 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b40b32640 1 -- 172.21.15.135:0/2961796460 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3c0b1b20 con 0x7f1b3c0b0860 2024-03-29T11:28:57.079 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b40b32640 1 -- 172.21.15.135:0/2961796460 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3c142da0 con 0x7f1b3c13ee40 2024-03-29T11:28:57.079 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b40b32640 1 -- 172.21.15.135:0/2961796460 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3c0b28f0 con 0x7f1b3c072450 2024-03-29T11:28:57.079 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b40b32640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:57.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b40b32640 10 monclient(hunting): authenticate will time out at 1044.453159s 2024-03-29T11:28:57.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b2affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:57.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b2affd640 10 client.?.objecter ms_handle_connect 0x7f1b3c13ee40 2024-03-29T11:28:57.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b2affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:57.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b2affd640 1 -- 172.21.15.135:0/2961796460 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3944592390 0 0) 0x7f1b3c0b1b20 con 0x7f1b3c0b0860 2024-03-29T11:28:57.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b2affd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:57.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b2affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:57.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:57.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b2affd640 10 monclient(hunting): my global_id is 4398 2024-03-29T11:28:57.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b2affd640 1 -- 172.21.15.135:0/2961796460 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b1c004eb0 con 0x7f1b3c0b0860 2024-03-29T11:28:57.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b2affd640 1 -- 172.21.15.135:0/2961796460 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2257386434 0 0) 0x7f1b3c142da0 con 0x7f1b3c13ee40 2024-03-29T11:28:57.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b2affd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:57.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b2affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:57.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:57.080 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b2affd640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:28:57.081 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b2affd640 1 -- 172.21.15.135:0/2961796460 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b3c0b1b20 con 0x7f1b3c13ee40 2024-03-29T11:28:57.081 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b2affd640 1 -- 172.21.15.135:0/2961796460 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2448886721 0 0) 0x7f1b1c004eb0 con 0x7f1b3c0b0860 2024-03-29T11:28:57.081 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b2affd640 1 -- 172.21.15.135:0/2961796460 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b3c142da0 con 0x7f1b3c0b0860 2024-03-29T11:28:57.081 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b2affd640 1 -- 172.21.15.135:0/2961796460 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f1b2c001bb0 con 0x7f1b3c0b0860 2024-03-29T11:28:57.081 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:57.081 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b2affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:57.081 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b2affd640 10 monclient(hunting): dump: 2024-03-29T11:28:57.081 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:57.081 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:57.081 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:57.081 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:57.081 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:57.081 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:57.081 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:57.082 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:57.082 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:57.082 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:57.082 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.084+0000 7f1b2affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:57.082 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.085+0000 7f1b2affd640 1 -- 172.21.15.135:0/2961796460 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3898273416 0 0) 0x7f1b3c142da0 con 0x7f1b3c0b0860 2024-03-29T11:28:57.082 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.085+0000 7f1b2affd640 1 -- 172.21.15.135:0/2961796460 >> v1:172.21.15.135:6790/0 conn(0x7f1b3c072450 legacy=0x7f1b3c0b3960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:57.085 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.087+0000 7f1b2affd640 1 -- 172.21.15.135:0/2961796460 >> v1:172.21.15.194:6789/0 conn(0x7f1b3c13ee40 legacy=0x7f1b3c1d35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.085 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.087+0000 7f1b2affd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:57.085 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.087+0000 7f1b2affd640 1 monclient: found mon.a 2024-03-29T11:28:57.085 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.087+0000 7f1b2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:57.085 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.087+0000 7f1b2affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:28:57.086 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.087+0000 7f1b2affd640 1 -- 172.21.15.135:0/2961796460 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3c0b2bd0 con 0x7f1b3c0b0860 2024-03-29T11:28:57.086 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.087+0000 7f1b2affd640 10 monclient: _finish_auth 0 2024-03-29T11:28:57.086 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.087+0000 7f1b2affd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:57.086 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.087+0000 7f1b2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:57.086 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.087+0000 7f1b40b32640 5 monclient: authenticate success, global_id 4398 2024-03-29T11:28:57.086 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.087+0000 7f1b40b32640 10 monclient: _renew_subs 2024-03-29T11:28:57.086 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.087+0000 7f1b40b32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:28:57.086 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.087+0000 7f1b40b32640 1 -- 172.21.15.135:0/2961796460 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b3c0b1dc0 con 0x7f1b3c0b0860 2024-03-29T11:28:57.086 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.087+0000 7f1b40b32640 10 client.4398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:57.086 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.087+0000 7f1b40b32640 10 monclient: _renew_subs 2024-03-29T11:28:57.086 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.087+0000 7f1b40b32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:28:57.086 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.087+0000 7f1b40b32640 1 -- 172.21.15.135:0/2961796460 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b3c143230 con 0x7f1b3c0b0860 2024-03-29T11:28:57.087 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.087+0000 7f1b40b32640 1 librados: init done 2024-03-29T11:28:57.087 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.087+0000 7f1b2affd640 1 -- 172.21.15.135:0/2961796460 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b2c001d10 con 0x7f1b3c0b0860 2024-03-29T11:28:57.087 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.087+0000 7f1b2affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:57.087 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.087+0000 7f1b2affd640 1 -- 172.21.15.135:0/2961796460 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f1b2c003be0 con 0x7f1b3c0b0860 2024-03-29T11:28:57.087 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.088+0000 7f1b2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:57.087 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.088+0000 7f1b2affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:57.087 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.088+0000 7f1b2affd640 10 monclient: dump: 2024-03-29T11:28:57.087 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:28:57.087 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:28:57.087 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:57.087 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:28:57.087 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:57.087 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:28:57.088 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:28:57.088 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:28:57.088 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:28:57.088 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:28:57.088 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.088+0000 7f1b38d99640 4 set_mon_vals no callback set 2024-03-29T11:28:57.088 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.089+0000 7f1b2affd640 1 -- 172.21.15.135:0/2961796460 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f1b2c003d40 con 0x7f1b3c0b0860 2024-03-29T11:28:57.089 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.089+0000 7f1b40b32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:57.089 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.089+0000 7f1b40b32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:57.090 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.089+0000 7f1b40b32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:28:57.090 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.089+0000 7f1b40b32640 1 -- 172.21.15.135:0/2961796460 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b20005080 con 0x7f1b3c0b0860 2024-03-29T11:28:57.090 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.090+0000 7f1b3a59c640 10 monclient: get_auth_request con 0x7f1b1c06daf0 auth_method 0 2024-03-29T11:28:57.090 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.090+0000 7f1b2affd640 1 -- 172.21.15.135:0/2961796460 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f1b2c084aa0 con 0x7f1b3c0b0860 2024-03-29T11:28:57.090 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.090+0000 7f1b2affd640 10 client.4398.objecter ms_dispatch 0x7f1b3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:57.090 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.090+0000 7f1b2affd640 3 client.4398.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:57.090 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.090+0000 7f1b2affd640 3 client.4398.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:57.090 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.090+0000 7f1b2affd640 20 client.4398.objecter dump_active .. 0 homeless 2024-03-29T11:28:57.090 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.090+0000 7f1b2affd640 10 client.4398.objecter ms_handle_connect 0x7f1b1c06daf0 2024-03-29T11:28:57.091 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.096+0000 7f1b2affd640 1 -- 172.21.15.135:0/2961796460 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b2c050380 con 0x7f1b3c0b0860 2024-03-29T11:28:57.091 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.097+0000 7f1b2affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:57.091 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.097+0000 7f1b2affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:57.390 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.399+0000 7f1b40b32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:57.391 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.399+0000 7f1b40b32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:57.391 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.399+0000 7f1b40b32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:28:57.391 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.399+0000 7f1b40b32640 1 -- 172.21.15.135:0/2961796460 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b20005330 con 0x7f1b3c0b0860 2024-03-29T11:28:57.391 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.400+0000 7f1b2affd640 1 -- 172.21.15.135:0/2961796460 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f1b2c0522b0 con 0x7f1b3c0b0860 2024-03-29T11:28:57.391 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.400+0000 7f1b2affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:57.391 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.400+0000 7f1b2affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:57.392 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:28:57.392 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:57.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.404+0000 7f1b40b32640 10 librados: watch_flush enter 2024-03-29T11:28:57.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.404+0000 7f1b40b32640 10 librados: watch_flush exit 2024-03-29T11:28:57.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.404+0000 7f1b40b32640 20 client.4398.objecter shutdown clearing up homeless session... 2024-03-29T11:28:57.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.404+0000 7f1b40b32640 10 client.4398.objecter successfully canceled tick 2024-03-29T11:28:57.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.404+0000 7f1b40b32640 1 -- 172.21.15.135:0/2961796460 >> v1:172.21.15.135:6816/3469875285 conn(0x7f1b1c06daf0 legacy=0x7f1b1c06ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.404+0000 7f1b40b32640 10 monclient: shutdown 2024-03-29T11:28:57.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.404+0000 7f1b40b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:57.396 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.404+0000 7f1b40b32640 1 -- 172.21.15.135:0/2961796460 >> v1:172.21.15.135:6789/0 conn(0x7f1b3c0b0860 legacy=0x7f1b3c1d2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:57.396 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.404+0000 7f1b40b32640 1 -- 172.21.15.135:0/2961796460 shutdown_connections 2024-03-29T11:28:57.396 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.404+0000 7f1b40b32640 1 -- 172.21.15.135:0/2961796460 >> 172.21.15.135:0/2961796460 conn(0x7f1b3c06b660 msgr2=0x7f1b3c0aa9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:57.396 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.404+0000 7f1b40b32640 1 -- 172.21.15.135:0/2961796460 shutdown_connections 2024-03-29T11:28:57.396 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.404+0000 7f1b40b32640 1 -- 172.21.15.135:0/2961796460 wait complete. 2024-03-29T11:28:57.396 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:28:57.404+0000 7f1b40b32640 1 librados: shutdown 2024-03-29T11:28:57.408 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:04.215 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:04.346 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.355+0000 7fddf859d640 10 monclient: get_monmap_and_config 2024-03-29T11:29:04.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.355+0000 7fddf859d640 10 monclient: build_initial_monmap 2024-03-29T11:29:04.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.355+0000 7fddf859d640 10 monclient: monmap: 2024-03-29T11:29:04.347 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:29:04.347 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:04.347 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:29:04.356723+0000 2024-03-29T11:29:04.347 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:29:04.356723+0000 2024-03-29T11:29:04.347 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:04.347 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:04.347 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:29:04.347 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:29:04.347 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:29:04.348 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:04.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.355+0000 7fddf859d640 1 Processor -- start 2024-03-29T11:29:04.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.355+0000 7fddf859d640 1 -- start start 2024-03-29T11:29:04.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.355+0000 7fddf859d640 10 monclient: init 2024-03-29T11:29:04.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.356+0000 7fddf859d640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:04.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.356+0000 7fddf859d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:04.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.356+0000 7fddf859d640 10 monclient(hunting): picked mon.noname-c con 0x7fddf01457f0 addr v1:172.21.15.194:6789/0 2024-03-29T11:29:04.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.357+0000 7fddf859d640 10 monclient(hunting): picked mon.noname-a con 0x7fddf013f650 addr v1:172.21.15.135:6789/0 2024-03-29T11:29:04.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.357+0000 7fddf859d640 10 monclient(hunting): picked mon.noname-b con 0x7fddf00a9d70 addr v1:172.21.15.135:6790/0 2024-03-29T11:29:04.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.357+0000 7fddf859d640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf00aaef0 con 0x7fddf013f650 2024-03-29T11:29:04.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.357+0000 7fddf859d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf0145bd0 con 0x7fddf01457f0 2024-03-29T11:29:04.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.357+0000 7fddf859d640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf0145eb0 con 0x7fddf00a9d70 2024-03-29T11:29:04.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.357+0000 7fddf859d640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:04.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.357+0000 7fddf859d640 10 monclient(hunting): authenticate will time out at 1051.726604s 2024-03-29T11:29:04.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.357+0000 7fddf6339640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fddf01457f0 0x7fddf013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:39152/0 (socket says 172.21.15.135:39152) 2024-03-29T11:29:04.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.357+0000 7fddf6339640 1 -- 172.21.15.135:0/3993778421 learned_addr learned my addr 172.21.15.135:0/3993778421 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:29:04.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.358+0000 7fddf5337640 1 -- 172.21.15.135:0/3993778421 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3364076688 0 0) 0x7fddf0145eb0 con 0x7fddf00a9d70 2024-03-29T11:29:04.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.358+0000 7fddf5337640 10 monclient(hunting): _negotiate 2024-03-29T11:29:04.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.358+0000 7fddf5337640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:04.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.358+0000 7fddf5337640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:04.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.358+0000 7fddf5337640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:29:04.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.358+0000 7fddf5337640 1 -- 172.21.15.135:0/3993778421 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdddc003520 con 0x7fddf00a9d70 2024-03-29T11:29:04.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.359+0000 7fddf5337640 1 -- 172.21.15.135:0/3993778421 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2358249809 0 0) 0x7fdddc003520 con 0x7fddf00a9d70 2024-03-29T11:29:04.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.359+0000 7fddf5337640 1 -- 172.21.15.135:0/3993778421 >> v1:172.21.15.194:6789/0 conn(0x7fddf01457f0 legacy=0x7fddf013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:04.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.359+0000 7fddf5337640 1 -- 172.21.15.135:0/3993778421 >> v1:172.21.15.135:6789/0 conn(0x7fddf013f650 legacy=0x7fddf00b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:04.365 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.361+0000 7fddf5337640 10 monclient: _finish_hunting 0 2024-03-29T11:29:04.365 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.361+0000 7fddf5337640 1 monclient: found mon.noname-b 2024-03-29T11:29:04.365 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.361+0000 7fddf5337640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:04.365 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.361+0000 7fddf5337640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:29:04.365 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.361+0000 7fddf5337640 1 -- 172.21.15.135:0/3993778421 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf0146340 con 0x7fddf00a9d70 2024-03-29T11:29:04.365 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.361+0000 7fddf5337640 10 monclient: _finish_auth 0 2024-03-29T11:29:04.365 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.361+0000 7fddf5337640 10 monclient: _check_auth_tickets 2024-03-29T11:29:04.366 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.361+0000 7fddf5337640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:04.366 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.361+0000 7fddf5337640 1 -- 172.21.15.135:0/3993778421 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fdde8001a10 con 0x7fddf00a9d70 2024-03-29T11:29:04.366 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.361+0000 7fddf5337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:04.366 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.361+0000 7fddf5337640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:04.366 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.361+0000 7fddf5337640 10 monclient: dump: 2024-03-29T11:29:04.366 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:04.366 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:04.366 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:04.366 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:04.366 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:04.366 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:04.366 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:04.366 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:04.367 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:04.367 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:04.367 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.361+0000 7fddf859d640 5 monclient: authenticate success, global_id 4391 2024-03-29T11:29:04.367 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.361+0000 7fddf859d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:04.367 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.361+0000 7fddf5337640 1 -- 172.21.15.135:0/3993778421 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdde8001ca0 con 0x7fddf00a9d70 2024-03-29T11:29:04.367 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.361+0000 7fddf5337640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:04.367 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.361+0000 7fddf5337640 1 -- 172.21.15.135:0/3993778421 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fdde8002030 con 0x7fddf00a9d70 2024-03-29T11:29:04.367 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.361+0000 7fddf5337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:04.367 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.361+0000 7fddf5337640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:04.367 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.361+0000 7fddf5337640 10 monclient: dump: 2024-03-29T11:29:04.367 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:04.368 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:04.368 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:04.368 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:04.368 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:04.368 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:04.368 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:04.368 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:04.368 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:04.368 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:04.368 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.361+0000 7fddf859d640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:04.368 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.361+0000 7fddf859d640 4 set_mon_vals no callback set 2024-03-29T11:29:04.368 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.362+0000 7fddf859d640 10 monclient: shutdown 2024-03-29T11:29:04.368 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.362+0000 7fddf859d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:04.368 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.362+0000 7fddf859d640 1 -- 172.21.15.135:0/3993778421 >> v1:172.21.15.135:6790/0 conn(0x7fddf00a9d70 legacy=0x7fddf014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:04.368 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.362+0000 7fddf859d640 1 -- 172.21.15.135:0/3993778421 shutdown_connections 2024-03-29T11:29:04.369 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.362+0000 7fddf859d640 1 -- 172.21.15.135:0/3993778421 >> 172.21.15.135:0/3993778421 conn(0x7fddf0066a00 msgr2=0x7fddf0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:04.369 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.362+0000 7fddf859d640 1 -- 172.21.15.135:0/3993778421 shutdown_connections 2024-03-29T11:29:04.369 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.362+0000 7fddf859d640 1 -- 172.21.15.135:0/3993778421 wait complete. 2024-03-29T11:29:04.369 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.363+0000 7fddf859d640 10 monclient: build_initial_monmap 2024-03-29T11:29:04.369 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.363+0000 7fddf859d640 10 monclient: monmap: 2024-03-29T11:29:04.369 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:29:04.369 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:04.369 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:29:04.369 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:29:04.369 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:04.369 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:04.370 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:29:04.370 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:29:04.370 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:29:04.370 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:04.370 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.363+0000 7fddf859d640 1 librados: starting msgr at 2024-03-29T11:29:04.370 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.363+0000 7fddf859d640 1 librados: starting objecter 2024-03-29T11:29:04.370 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.363+0000 7fddf859d640 1 Processor -- start 2024-03-29T11:29:04.370 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.363+0000 7fddf859d640 1 -- start start 2024-03-29T11:29:04.370 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.363+0000 7fddf859d640 1 librados: setting wanted keys 2024-03-29T11:29:04.370 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.363+0000 7fddf859d640 1 librados: calling monclient init 2024-03-29T11:29:04.370 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.363+0000 7fddf859d640 10 monclient: init 2024-03-29T11:29:04.370 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.364+0000 7fddf859d640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:04.370 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.364+0000 7fddf859d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:04.370 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.364+0000 7fddf859d640 10 monclient(hunting): picked mon.noname-b con 0x7fddf00a9d70 addr v1:172.21.15.194:6789/0 2024-03-29T11:29:04.371 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.364+0000 7fddf859d640 10 monclient(hunting): picked mon.noname-a con 0x7fddf013f650 addr v1:172.21.15.135:6789/0 2024-03-29T11:29:04.371 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.364+0000 7fddf859d640 10 monclient(hunting): picked mon.noname-c con 0x7fddf01457f0 addr v1:172.21.15.135:6790/0 2024-03-29T11:29:04.371 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.364+0000 7fddf859d640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf0142d20 con 0x7fddf013f650 2024-03-29T11:29:04.371 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.364+0000 7fddf859d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf0143060 con 0x7fddf00a9d70 2024-03-29T11:29:04.371 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.365+0000 7fddf859d640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf0135b80 con 0x7fddf01457f0 2024-03-29T11:29:04.371 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.365+0000 7fddf859d640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:04.371 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.365+0000 7fddf859d640 10 monclient(hunting): authenticate will time out at 1051.734264s 2024-03-29T11:29:04.371 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.365+0000 7fddf6339640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fddf00a9d70 0x7fddf0140010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:39360/0 (socket says 172.21.15.135:39360) 2024-03-29T11:29:04.371 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.365+0000 7fddf6339640 1 -- 172.21.15.135:0/3286580870 learned_addr learned my addr 172.21.15.135:0/3286580870 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:29:04.372 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.365+0000 7fdde6ffd640 10 client.?.objecter ms_handle_connect 0x7fddf00a9d70 2024-03-29T11:29:04.372 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.365+0000 7fdde6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:04.372 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.365+0000 7fdde6ffd640 10 client.?.objecter ms_handle_connect 0x7fddf01457f0 2024-03-29T11:29:04.372 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.365+0000 7fdde6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:04.372 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.365+0000 7fdde6ffd640 1 -- 172.21.15.135:0/3286580870 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888435023 0 0) 0x7fddf0135b80 con 0x7fddf01457f0 2024-03-29T11:29:04.372 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.366+0000 7fdde6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:04.372 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.366+0000 7fdde6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:04.372 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.366+0000 7fdde6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:04.372 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.366+0000 7fdde6ffd640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:29:04.372 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.366+0000 7fdde6ffd640 1 -- 172.21.15.135:0/3286580870 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddd4004eb0 con 0x7fddf01457f0 2024-03-29T11:29:04.373 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.366+0000 7fdde6ffd640 1 -- 172.21.15.135:0/3286580870 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3655122127 0 0) 0x7fddd4004eb0 con 0x7fddf01457f0 2024-03-29T11:29:04.373 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.366+0000 7fdde6ffd640 1 -- 172.21.15.135:0/3286580870 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddf0135b80 con 0x7fddf01457f0 2024-03-29T11:29:04.373 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.366+0000 7fdde6ffd640 1 -- 172.21.15.135:0/3286580870 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fdde80018d0 con 0x7fddf01457f0 2024-03-29T11:29:04.373 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.366+0000 7fdde6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:04.373 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.366+0000 7fdde6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:04.373 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.366+0000 7fdde6ffd640 10 monclient(hunting): dump: 2024-03-29T11:29:04.373 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:04.373 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:04.373 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:04.373 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:04.373 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:04.374 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:04.374 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:04.374 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:04.374 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:04.374 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:04.374 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.366+0000 7fdde6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:04.374 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.366+0000 7fdde6ffd640 1 -- 172.21.15.135:0/3286580870 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 855693031 0 0) 0x7fddf0135b80 con 0x7fddf01457f0 2024-03-29T11:29:04.374 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.366+0000 7fdde6ffd640 1 -- 172.21.15.135:0/3286580870 >> v1:172.21.15.194:6789/0 conn(0x7fddf00a9d70 legacy=0x7fddf0140010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:04.374 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.366+0000 7fdde6ffd640 1 -- 172.21.15.135:0/3286580870 >> v1:172.21.15.135:6789/0 conn(0x7fddf013f650 legacy=0x7fddf0140bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:04.374 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.367+0000 7fdde6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:04.374 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.367+0000 7fdde6ffd640 1 monclient: found mon.c 2024-03-29T11:29:04.375 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.367+0000 7fdde6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:04.375 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.367+0000 7fdde6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:29:04.375 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.367+0000 7fdde6ffd640 1 -- 172.21.15.135:0/3286580870 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf0135d10 con 0x7fddf01457f0 2024-03-29T11:29:04.375 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.367+0000 7fdde6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:29:04.375 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.367+0000 7fdde6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:04.375 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.367+0000 7fdde6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:04.375 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.368+0000 7fdde6ffd640 1 -- 172.21.15.135:0/3286580870 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdde8003ce0 con 0x7fddf01457f0 2024-03-29T11:29:04.375 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.368+0000 7fdde6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:04.375 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.368+0000 7fdde6ffd640 1 -- 172.21.15.135:0/3286580870 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fdde8003ff0 con 0x7fddf01457f0 2024-03-29T11:29:04.375 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.368+0000 7fdde6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:04.375 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.368+0000 7fdde6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:04.375 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.368+0000 7fdde6ffd640 10 monclient: dump: 2024-03-29T11:29:04.375 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:04.375 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:04.376 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:04.376 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:04.376 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:04.376 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:04.376 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:04.376 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:04.376 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:04.376 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:04.376 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.368+0000 7fddf859d640 5 monclient: authenticate success, global_id 4394 2024-03-29T11:29:04.376 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.368+0000 7fddf859d640 10 monclient: _renew_subs 2024-03-29T11:29:04.376 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.368+0000 7fddf859d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:29:04.376 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.368+0000 7fddf859d640 1 -- 172.21.15.135:0/3286580870 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf0142840 con 0x7fddf01457f0 2024-03-29T11:29:04.376 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.368+0000 7fddf4b36640 4 set_mon_vals no callback set 2024-03-29T11:29:04.376 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.369+0000 7fddf859d640 10 client.4394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:04.376 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.369+0000 7fddf859d640 10 monclient: _renew_subs 2024-03-29T11:29:04.376 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.369+0000 7fddf859d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:29:04.377 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.369+0000 7fddf859d640 1 -- 172.21.15.135:0/3286580870 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf01414b0 con 0x7fddf01457f0 2024-03-29T11:29:04.377 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.369+0000 7fddf859d640 1 librados: init done 2024-03-29T11:29:04.377 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.370+0000 7fdde4ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:04.377 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.370+0000 7fdde4ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:04.377 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.370+0000 7fdde4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:29:04.377 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.370+0000 7fdde4ff9640 1 -- 172.21.15.135:0/3286580870 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddcc005080 con 0x7fddf01457f0 2024-03-29T11:29:04.377 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.370+0000 7fdde6ffd640 1 -- 172.21.15.135:0/3286580870 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fdde8019fc0 con 0x7fddf01457f0 2024-03-29T11:29:04.377 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.371+0000 7fdde6ffd640 1 -- 172.21.15.135:0/3286580870 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7fdde8083cd0 con 0x7fddf01457f0 2024-03-29T11:29:04.377 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.371+0000 7fdde6ffd640 10 client.4394.objecter ms_dispatch 0x7fddf0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:04.377 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.371+0000 7fdde6ffd640 3 client.4394.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:04.377 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.371+0000 7fdde6ffd640 3 client.4394.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:04.377 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.371+0000 7fdde6ffd640 20 client.4394.objecter dump_active .. 0 homeless 2024-03-29T11:29:04.377 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.371+0000 7fddf6339640 10 monclient: get_auth_request con 0x7fddd406dcd0 auth_method 0 2024-03-29T11:29:04.377 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.371+0000 7fdde6ffd640 10 client.4394.objecter ms_handle_connect 0x7fddd406dcd0 2024-03-29T11:29:04.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.377+0000 7fdde6ffd640 1 -- 172.21.15.135:0/3286580870 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdde804f6f0 con 0x7fddf01457f0 2024-03-29T11:29:04.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.377+0000 7fdde6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:04.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.377+0000 7fdde6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:04.684 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.693+0000 7fdde4ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:04.684 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.693+0000 7fdde4ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:04.684 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.693+0000 7fdde4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:29:04.684 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.693+0000 7fdde4ff9640 1 -- 172.21.15.135:0/3286580870 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fddcc005330 con 0x7fddf01457f0 2024-03-29T11:29:04.685 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.693+0000 7fdde6ffd640 1 -- 172.21.15.135:0/3286580870 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7fdde8051620 con 0x7fddf01457f0 2024-03-29T11:29:04.685 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.694+0000 7fdde6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:04.685 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.694+0000 7fdde6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:04.686 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:29:04.686 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:04.688 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.697+0000 7fdde4ff9640 10 librados: watch_flush enter 2024-03-29T11:29:04.690 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.697+0000 7fdde4ff9640 10 librados: watch_flush exit 2024-03-29T11:29:04.690 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.697+0000 7fdde4ff9640 20 client.4394.objecter shutdown clearing up homeless session... 2024-03-29T11:29:04.690 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.697+0000 7fdde4ff9640 10 client.4394.objecter successfully canceled tick 2024-03-29T11:29:04.690 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.698+0000 7fdde4ff9640 1 -- 172.21.15.135:0/3286580870 >> v1:172.21.15.135:6816/3469875285 conn(0x7fddd406dcd0 legacy=0x7fddd4070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:04.690 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.698+0000 7fdde4ff9640 10 monclient: shutdown 2024-03-29T11:29:04.690 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.698+0000 7fdde4ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:04.690 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.698+0000 7fdde4ff9640 1 -- 172.21.15.135:0/3286580870 >> v1:172.21.15.135:6790/0 conn(0x7fddf01457f0 legacy=0x7fddf01e1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:04.691 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.698+0000 7fdde4ff9640 1 -- 172.21.15.135:0/3286580870 shutdown_connections 2024-03-29T11:29:04.691 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.698+0000 7fdde4ff9640 1 -- 172.21.15.135:0/3286580870 >> 172.21.15.135:0/3286580870 conn(0x7fddf0066a00 msgr2=0x7fddf013b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:04.691 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.698+0000 7fdde4ff9640 1 -- 172.21.15.135:0/3286580870 shutdown_connections 2024-03-29T11:29:04.691 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.698+0000 7fdde4ff9640 1 -- 172.21.15.135:0/3286580870 wait complete. 2024-03-29T11:29:04.691 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:04.698+0000 7fdde4ff9640 1 librados: shutdown 2024-03-29T11:29:04.706 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:11.513 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:11.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.673+0000 7fdd52ef6640 10 monclient: get_monmap_and_config 2024-03-29T11:29:11.665 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.673+0000 7fdd52ef6640 10 monclient: build_initial_monmap 2024-03-29T11:29:11.665 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.673+0000 7fdd52ef6640 10 monclient: monmap: 2024-03-29T11:29:11.665 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:29:11.665 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:11.665 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:29:11.674557+0000 2024-03-29T11:29:11.665 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:29:11.674557+0000 2024-03-29T11:29:11.665 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:11.665 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:11.665 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:29:11.665 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:29:11.665 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:29:11.665 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:11.666 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.675+0000 7fdd52ef6640 1 Processor -- start 2024-03-29T11:29:11.666 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.675+0000 7fdd52ef6640 1 -- start start 2024-03-29T11:29:11.666 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.675+0000 7fdd52ef6640 10 monclient: init 2024-03-29T11:29:11.667 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.676+0000 7fdd52ef6640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:11.667 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.676+0000 7fdd52ef6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:11.667 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.676+0000 7fdd52ef6640 10 monclient(hunting): picked mon.noname-a con 0x7fdd4c0aeda0 addr v1:172.21.15.135:6789/0 2024-03-29T11:29:11.667 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.676+0000 7fdd52ef6640 10 monclient(hunting): picked mon.noname-b con 0x7fdd4c13bd60 addr v1:172.21.15.135:6790/0 2024-03-29T11:29:11.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.676+0000 7fdd52ef6640 10 monclient(hunting): picked mon.noname-c con 0x7fdd4c071790 addr v1:172.21.15.194:6789/0 2024-03-29T11:29:11.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.676+0000 7fdd52ef6640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd4c06f550 con 0x7fdd4c0aeda0 2024-03-29T11:29:11.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.676+0000 7fdd52ef6640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd4c072330 con 0x7fdd4c071790 2024-03-29T11:29:11.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.676+0000 7fdd52ef6640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd4c06fd20 con 0x7fdd4c13bd60 2024-03-29T11:29:11.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.676+0000 7fdd52ef6640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:11.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.676+0000 7fdd52ef6640 10 monclient(hunting): authenticate will time out at 1059.044733s 2024-03-29T11:29:11.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.678+0000 7fdd4bfff640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7fdd4c13bd60 0x7fdd4c13c1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:51818/0 (socket says 172.21.15.135:51818) 2024-03-29T11:29:11.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.678+0000 7fdd4bfff640 1 -- 172.21.15.135:0/849970417 learned_addr learned my addr 172.21.15.135:0/849970417 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:29:11.669 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.678+0000 7fdd4b7fe640 1 -- 172.21.15.135:0/849970417 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1276229916 0 0) 0x7fdd4c06fd20 con 0x7fdd4c13bd60 2024-03-29T11:29:11.669 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.678+0000 7fdd4b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:11.669 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.678+0000 7fdd4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:11.669 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.678+0000 7fdd4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:11.669 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.678+0000 7fdd4b7fe640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:29:11.669 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.678+0000 7fdd4b7fe640 1 -- 172.21.15.135:0/849970417 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd34003520 con 0x7fdd4c13bd60 2024-03-29T11:29:11.669 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.678+0000 7fdd4b7fe640 1 -- 172.21.15.135:0/849970417 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3673453145 0 0) 0x7fdd34003520 con 0x7fdd4c13bd60 2024-03-29T11:29:11.669 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.678+0000 7fdd4b7fe640 1 -- 172.21.15.135:0/849970417 >> v1:172.21.15.194:6789/0 conn(0x7fdd4c071790 legacy=0x7fdd4c071c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:11.675 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.679+0000 7fdd4b7fe640 1 -- 172.21.15.135:0/849970417 >> v1:172.21.15.135:6789/0 conn(0x7fdd4c0aeda0 legacy=0x7fdd4c06ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:11.675 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.679+0000 7fdd4b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:11.676 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.679+0000 7fdd4b7fe640 1 monclient: found mon.noname-b 2024-03-29T11:29:11.676 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.679+0000 7fdd4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:11.676 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.679+0000 7fdd4b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:29:11.676 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.679+0000 7fdd4b7fe640 1 -- 172.21.15.135:0/849970417 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd4c145620 con 0x7fdd4c13bd60 2024-03-29T11:29:11.676 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.679+0000 7fdd4b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:11.676 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.679+0000 7fdd4b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:11.677 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.679+0000 7fdd4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:11.677 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.679+0000 7fdd4b7fe640 1 -- 172.21.15.135:0/849970417 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fdd44001a10 con 0x7fdd4c13bd60 2024-03-29T11:29:11.677 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.679+0000 7fdd4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:11.677 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.679+0000 7fdd4b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:11.677 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.679+0000 7fdd4b7fe640 10 monclient: dump: 2024-03-29T11:29:11.677 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:11.678 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:11.678 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:11.678 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:11.678 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:11.678 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:11.679 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:11.679 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:11.679 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:11.680 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:11.680 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.680+0000 7fdd4b7fe640 1 -- 172.21.15.135:0/849970417 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd44001c70 con 0x7fdd4c13bd60 2024-03-29T11:29:11.680 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.680+0000 7fdd4b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:11.680 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.680+0000 7fdd4b7fe640 1 -- 172.21.15.135:0/849970417 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fdd44002040 con 0x7fdd4c13bd60 2024-03-29T11:29:11.680 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.680+0000 7fdd4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:11.680 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.680+0000 7fdd4b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:11.680 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.680+0000 7fdd4b7fe640 10 monclient: dump: 2024-03-29T11:29:11.681 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:11.681 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:11.681 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:11.681 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:11.681 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:11.681 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:11.681 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:11.681 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:11.681 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:11.682 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:11.682 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.680+0000 7fdd52ef6640 5 monclient: authenticate success, global_id 4397 2024-03-29T11:29:11.682 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.680+0000 7fdd52ef6640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:11.682 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.680+0000 7fdd52ef6640 4 set_mon_vals no callback set 2024-03-29T11:29:11.682 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.681+0000 7fdd52ef6640 10 monclient: shutdown 2024-03-29T11:29:11.682 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.681+0000 7fdd52ef6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:11.682 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.681+0000 7fdd52ef6640 1 -- 172.21.15.135:0/849970417 >> v1:172.21.15.135:6790/0 conn(0x7fdd4c13bd60 legacy=0x7fdd4c13c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:11.682 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.681+0000 7fdd52ef6640 1 -- 172.21.15.135:0/849970417 shutdown_connections 2024-03-29T11:29:11.682 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.682+0000 7fdd52ef6640 1 -- 172.21.15.135:0/849970417 >> 172.21.15.135:0/849970417 conn(0x7fdd4c067470 msgr2=0x7fdd4c0b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:11.683 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.682+0000 7fdd52ef6640 1 -- 172.21.15.135:0/849970417 shutdown_connections 2024-03-29T11:29:11.683 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.682+0000 7fdd52ef6640 1 -- 172.21.15.135:0/849970417 wait complete. 2024-03-29T11:29:11.683 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.683+0000 7fdd52ef6640 10 monclient: build_initial_monmap 2024-03-29T11:29:11.683 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.683+0000 7fdd52ef6640 10 monclient: monmap: 2024-03-29T11:29:11.683 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:29:11.683 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:11.683 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:29:11.683 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:29:11.684 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:11.684 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:11.684 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:29:11.684 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:29:11.684 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:29:11.684 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:11.684 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.683+0000 7fdd52ef6640 1 librados: starting msgr at 2024-03-29T11:29:11.685 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.683+0000 7fdd52ef6640 1 librados: starting objecter 2024-03-29T11:29:11.685 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.683+0000 7fdd52ef6640 1 Processor -- start 2024-03-29T11:29:11.685 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.684+0000 7fdd52ef6640 1 -- start start 2024-03-29T11:29:11.685 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.684+0000 7fdd52ef6640 1 librados: setting wanted keys 2024-03-29T11:29:11.685 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.684+0000 7fdd52ef6640 1 librados: calling monclient init 2024-03-29T11:29:11.685 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.684+0000 7fdd52ef6640 10 monclient: init 2024-03-29T11:29:11.691 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.685+0000 7fdd52ef6640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:11.691 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.685+0000 7fdd52ef6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:11.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.685+0000 7fdd52ef6640 10 monclient(hunting): picked mon.noname-a con 0x7fdd4c071790 addr v1:172.21.15.135:6789/0 2024-03-29T11:29:11.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.685+0000 7fdd52ef6640 10 monclient(hunting): picked mon.noname-c con 0x7fdd4c0aeda0 addr v1:172.21.15.135:6790/0 2024-03-29T11:29:11.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.685+0000 7fdd52ef6640 10 monclient(hunting): picked mon.noname-b con 0x7fdd4c13bd60 addr v1:172.21.15.194:6789/0 2024-03-29T11:29:11.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.685+0000 7fdd52ef6640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd4c13de20 con 0x7fdd4c071790 2024-03-29T11:29:11.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.685+0000 7fdd52ef6640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd4c13f860 con 0x7fdd4c13bd60 2024-03-29T11:29:11.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.685+0000 7fdd52ef6640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd4c13d1f0 con 0x7fdd4c0aeda0 2024-03-29T11:29:11.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.685+0000 7fdd52ef6640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:11.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.685+0000 7fdd52ef6640 10 monclient(hunting): authenticate will time out at 1059.053494s 2024-03-29T11:29:11.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.687+0000 7fdd4bfff640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7fdd4c0aeda0 0x7fdd4c1ebab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:53784/0 (socket says 172.21.15.135:53784) 2024-03-29T11:29:11.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.687+0000 7fdd4bfff640 1 -- 172.21.15.135:0/3182298478 learned_addr learned my addr 172.21.15.135:0/3182298478 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:29:11.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.688+0000 7fdd497fa640 10 client.?.objecter ms_handle_connect 0x7fdd4c0aeda0 2024-03-29T11:29:11.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.688+0000 7fdd497fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:11.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.689+0000 7fdd497fa640 1 -- 172.21.15.135:0/3182298478 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588332681 0 0) 0x7fdd4c13d1f0 con 0x7fdd4c0aeda0 2024-03-29T11:29:11.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.689+0000 7fdd497fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:11.692 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.689+0000 7fdd497fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:11.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.689+0000 7fdd497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:11.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.689+0000 7fdd497fa640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:29:11.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.689+0000 7fdd497fa640 1 -- 172.21.15.135:0/3182298478 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd38004eb0 con 0x7fdd4c0aeda0 2024-03-29T11:29:11.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.691+0000 7fdd497fa640 1 -- 172.21.15.135:0/3182298478 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3614933289 0 0) 0x7fdd38004eb0 con 0x7fdd4c0aeda0 2024-03-29T11:29:11.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.691+0000 7fdd497fa640 1 -- 172.21.15.135:0/3182298478 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd4c13d1f0 con 0x7fdd4c0aeda0 2024-03-29T11:29:11.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.691+0000 7fdd497fa640 1 -- 172.21.15.135:0/3182298478 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fdd44002040 con 0x7fdd4c0aeda0 2024-03-29T11:29:11.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.691+0000 7fdd497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:11.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.691+0000 7fdd497fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:11.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.691+0000 7fdd497fa640 10 monclient(hunting): dump: 2024-03-29T11:29:11.693 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:11.693 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:11.693 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:11.693 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:11.693 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:11.694 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:11.694 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:11.694 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:11.694 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:11.694 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:11.694 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.691+0000 7fdd497fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:11.694 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.691+0000 7fdd497fa640 10 client.?.objecter ms_handle_connect 0x7fdd4c13bd60 2024-03-29T11:29:11.694 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.691+0000 7fdd497fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:11.694 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.692+0000 7fdd497fa640 1 -- 172.21.15.135:0/3182298478 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2190392126 0 0) 0x7fdd4c13d1f0 con 0x7fdd4c0aeda0 2024-03-29T11:29:11.694 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.692+0000 7fdd497fa640 1 -- 172.21.15.135:0/3182298478 >> v1:172.21.15.194:6789/0 conn(0x7fdd4c13bd60 legacy=0x7fdd4c1ef1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:11.694 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.692+0000 7fdd497fa640 1 -- 172.21.15.135:0/3182298478 >> v1:172.21.15.135:6789/0 conn(0x7fdd4c071790 legacy=0x7fdd4c13d710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:11.694 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.693+0000 7fdd497fa640 10 monclient: _finish_hunting 0 2024-03-29T11:29:11.694 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.693+0000 7fdd497fa640 1 monclient: found mon.c 2024-03-29T11:29:11.694 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.693+0000 7fdd497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.693+0000 7fdd497fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.693+0000 7fdd497fa640 1 -- 172.21.15.135:0/3182298478 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd4c13fb40 con 0x7fdd4c0aeda0 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.693+0000 7fdd497fa640 10 monclient: _finish_auth 0 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.693+0000 7fdd497fa640 10 monclient: _check_auth_tickets 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.693+0000 7fdd497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.693+0000 7fdd52ef6640 5 monclient: authenticate success, global_id 4400 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.693+0000 7fdd52ef6640 10 monclient: _renew_subs 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.693+0000 7fdd52ef6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.693+0000 7fdd52ef6640 1 -- 172.21.15.135:0/3182298478 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd4c137a10 con 0x7fdd4c0aeda0 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.693+0000 7fdd52ef6640 10 client.4400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.693+0000 7fdd52ef6640 10 monclient: _renew_subs 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.693+0000 7fdd52ef6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.693+0000 7fdd52ef6640 1 -- 172.21.15.135:0/3182298478 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd4c13cd00 con 0x7fdd4c0aeda0 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.693+0000 7fdd52ef6640 1 librados: init done 2024-03-29T11:29:11.695 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.693+0000 7fdd497fa640 1 -- 172.21.15.135:0/3182298478 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd44004e80 con 0x7fdd4c0aeda0 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.693+0000 7fdd497fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.693+0000 7fdd4affd640 4 set_mon_vals no callback set 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.694+0000 7fdd497fa640 1 -- 172.21.15.135:0/3182298478 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fdd44005250 con 0x7fdd4c0aeda0 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.694+0000 7fdd497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.694+0000 7fdd497fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.694+0000 7fdd497fa640 10 monclient: dump: 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:11.696 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:11.697 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.695+0000 7fdd497fa640 1 -- 172.21.15.135:0/3182298478 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fdd440053b0 con 0x7fdd4c0aeda0 2024-03-29T11:29:11.697 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.696+0000 7fdd497fa640 1 -- 172.21.15.135:0/3182298478 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7fdd44085010 con 0x7fdd4c0aeda0 2024-03-29T11:29:11.697 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.696+0000 7fdd497fa640 10 client.4400.objecter ms_dispatch 0x7fdd4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:11.697 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.696+0000 7fdd497fa640 3 client.4400.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:11.697 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.696+0000 7fdd497fa640 3 client.4400.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:11.697 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.696+0000 7fdd52ef6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:11.697 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.696+0000 7fdd52ef6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:11.697 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.696+0000 7fdd52ef6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:29:11.697 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.696+0000 7fdd52ef6640 1 -- 172.21.15.135:0/3182298478 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd2c005080 con 0x7fdd4c0aeda0 2024-03-29T11:29:11.697 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.697+0000 7fdd497fa640 20 client.4400.objecter dump_active .. 0 homeless 2024-03-29T11:29:11.697 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.699+0000 7fdd50c92640 10 monclient: get_auth_request con 0x7fdd3806dc60 auth_method 0 2024-03-29T11:29:11.697 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.700+0000 7fdd497fa640 10 client.4400.objecter ms_handle_connect 0x7fdd3806dc60 2024-03-29T11:29:11.698 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.705+0000 7fdd497fa640 1 -- 172.21.15.135:0/3182298478 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdd440508f0 con 0x7fdd4c0aeda0 2024-03-29T11:29:11.698 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.705+0000 7fdd497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:11.698 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:11.705+0000 7fdd497fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:12.026 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:12.035+0000 7fdd52ef6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:12.026 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:12.036+0000 7fdd52ef6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:12.026 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:12.036+0000 7fdd52ef6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:29:12.026 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:12.036+0000 7fdd52ef6640 1 -- 172.21.15.135:0/3182298478 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd2c005330 con 0x7fdd4c0aeda0 2024-03-29T11:29:12.029 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:12.038+0000 7fdd497fa640 1 -- 172.21.15.135:0/3182298478 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7fdd44052820 con 0x7fdd4c0aeda0 2024-03-29T11:29:12.029 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:12.039+0000 7fdd497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:12.029 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:12.039+0000 7fdd497fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:12.029 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:29:12.030 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:12.034 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:12.043+0000 7fdd52ef6640 10 librados: watch_flush enter 2024-03-29T11:29:12.034 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:12.044+0000 7fdd52ef6640 10 librados: watch_flush exit 2024-03-29T11:29:12.034 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:12.044+0000 7fdd52ef6640 20 client.4400.objecter shutdown clearing up homeless session... 2024-03-29T11:29:12.034 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:12.044+0000 7fdd52ef6640 10 client.4400.objecter successfully canceled tick 2024-03-29T11:29:12.036 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:12.046+0000 7fdd52ef6640 1 -- 172.21.15.135:0/3182298478 >> v1:172.21.15.135:6816/3469875285 conn(0x7fdd3806dc60 legacy=0x7fdd38070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:12.036 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:12.046+0000 7fdd52ef6640 10 monclient: shutdown 2024-03-29T11:29:12.036 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:12.046+0000 7fdd52ef6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:12.036 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:12.046+0000 7fdd52ef6640 1 -- 172.21.15.135:0/3182298478 >> v1:172.21.15.135:6790/0 conn(0x7fdd4c0aeda0 legacy=0x7fdd4c1ebab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:12.036 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:12.046+0000 7fdd52ef6640 1 -- 172.21.15.135:0/3182298478 shutdown_connections 2024-03-29T11:29:12.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:12.046+0000 7fdd52ef6640 1 -- 172.21.15.135:0/3182298478 >> 172.21.15.135:0/3182298478 conn(0x7fdd4c067470 msgr2=0x7fdd4c06ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:12.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:12.047+0000 7fdd52ef6640 1 -- 172.21.15.135:0/3182298478 shutdown_connections 2024-03-29T11:29:12.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:12.047+0000 7fdd52ef6640 1 -- 172.21.15.135:0/3182298478 wait complete. 2024-03-29T11:29:12.037 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:12.047+0000 7fdd52ef6640 1 librados: shutdown 2024-03-29T11:29:12.055 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:18.863 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:19.021 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.026+0000 7f1d700b3640 10 monclient: get_monmap_and_config 2024-03-29T11:29:19.021 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.026+0000 7f1d700b3640 10 monclient: build_initial_monmap 2024-03-29T11:29:19.021 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.026+0000 7f1d700b3640 10 monclient: monmap: 2024-03-29T11:29:19.021 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:29:19.021 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:19.021 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:29:19.027284+0000 2024-03-29T11:29:19.021 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:29:19.027284+0000 2024-03-29T11:29:19.021 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:19.021 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:19.021 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:29:19.021 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:29:19.021 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:29:19.022 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:19.022 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.027+0000 7f1d700b3640 1 Processor -- start 2024-03-29T11:29:19.022 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.028+0000 7f1d700b3640 1 -- start start 2024-03-29T11:29:19.022 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.028+0000 7f1d700b3640 10 monclient: init 2024-03-29T11:29:19.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.028+0000 7f1d700b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:19.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.028+0000 7f1d700b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:19.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.028+0000 7f1d700b3640 10 monclient(hunting): picked mon.noname-c con 0x7f1d680a9750 addr v1:172.21.15.194:6789/0 2024-03-29T11:29:19.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.028+0000 7f1d700b3640 10 monclient(hunting): picked mon.noname-b con 0x7f1d68149ac0 addr v1:172.21.15.135:6790/0 2024-03-29T11:29:19.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.028+0000 7f1d700b3640 10 monclient(hunting): picked mon.noname-a con 0x7f1d6814fe50 addr v1:172.21.15.135:6789/0 2024-03-29T11:29:19.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.028+0000 7f1d700b3640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d680aa4a0 con 0x7f1d6814fe50 2024-03-29T11:29:19.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.028+0000 7f1d700b3640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d6814a680 con 0x7f1d680a9750 2024-03-29T11:29:19.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.028+0000 7f1d700b3640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d680af290 con 0x7f1d68149ac0 2024-03-29T11:29:19.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.028+0000 7f1d700b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:19.025 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.028+0000 7f1d700b3640 10 monclient(hunting): authenticate will time out at 1066.397015s 2024-03-29T11:29:19.030 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.035+0000 7f1d6d64e640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f1d68149ac0 0x7f1d68149f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:57759/0 (socket says 172.21.15.135:57759) 2024-03-29T11:29:19.030 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.035+0000 7f1d6d64e640 1 -- 172.21.15.135:0/1982882616 learned_addr learned my addr 172.21.15.135:0/1982882616 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:29:19.030 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.035+0000 7f1d6ce4d640 1 -- 172.21.15.135:0/1982882616 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1867978687 0 0) 0x7f1d680af290 con 0x7f1d68149ac0 2024-03-29T11:29:19.030 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.035+0000 7f1d6ce4d640 10 monclient(hunting): _negotiate 2024-03-29T11:29:19.030 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.035+0000 7f1d6ce4d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:19.030 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.035+0000 7f1d6ce4d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:19.030 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.035+0000 7f1d6ce4d640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:29:19.030 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.035+0000 7f1d6ce4d640 1 -- 172.21.15.135:0/1982882616 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d54003520 con 0x7f1d68149ac0 2024-03-29T11:29:19.030 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.036+0000 7f1d6ce4d640 1 -- 172.21.15.135:0/1982882616 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2570715420 0 0) 0x7f1d54003520 con 0x7f1d68149ac0 2024-03-29T11:29:19.031 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.036+0000 7f1d6ce4d640 1 -- 172.21.15.135:0/1982882616 >> v1:172.21.15.194:6789/0 conn(0x7f1d680a9750 legacy=0x7f1d681436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:19.031 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.036+0000 7f1d6ce4d640 1 -- 172.21.15.135:0/1982882616 >> v1:172.21.15.135:6789/0 conn(0x7f1d6814fe50 legacy=0x7f1d68152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:19.031 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.036+0000 7f1d6ce4d640 10 monclient: _finish_hunting 0 2024-03-29T11:29:19.031 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.036+0000 7f1d6ce4d640 1 monclient: found mon.noname-b 2024-03-29T11:29:19.031 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.036+0000 7f1d6ce4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:19.031 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.036+0000 7f1d6ce4d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:29:19.031 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.036+0000 7f1d6ce4d640 1 -- 172.21.15.135:0/1982882616 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d680afb40 con 0x7f1d68149ac0 2024-03-29T11:29:19.031 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.036+0000 7f1d6ce4d640 10 monclient: _finish_auth 0 2024-03-29T11:29:19.031 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.036+0000 7f1d6ce4d640 10 monclient: _check_auth_tickets 2024-03-29T11:29:19.031 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.036+0000 7f1d6ce4d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:19.031 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.036+0000 7f1d6ce4d640 1 -- 172.21.15.135:0/1982882616 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f1d58001690 con 0x7f1d68149ac0 2024-03-29T11:29:19.031 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.036+0000 7f1d6ce4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:19.031 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.036+0000 7f1d6ce4d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:19.032 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.036+0000 7f1d6ce4d640 10 monclient: dump: 2024-03-29T11:29:19.032 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:19.032 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:19.032 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:19.032 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:19.032 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:19.032 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:19.032 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:19.032 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:19.032 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:19.032 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:19.032 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.037+0000 7f1d700b3640 5 monclient: authenticate success, global_id 4403 2024-03-29T11:29:19.032 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.037+0000 7f1d700b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:19.033 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.040+0000 7f1d6ce4d640 1 -- 172.21.15.135:0/1982882616 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d580017f0 con 0x7f1d68149ac0 2024-03-29T11:29:19.033 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.040+0000 7f1d6ce4d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:19.033 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.040+0000 7f1d6ce4d640 1 -- 172.21.15.135:0/1982882616 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f1d58001c40 con 0x7f1d68149ac0 2024-03-29T11:29:19.033 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.040+0000 7f1d6ce4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:19.033 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.040+0000 7f1d6ce4d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:19.033 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.040+0000 7f1d6ce4d640 10 monclient: dump: 2024-03-29T11:29:19.033 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:19.033 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:19.034 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:19.034 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:19.034 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:19.034 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:19.034 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:19.034 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:19.034 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:19.034 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:19.034 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.040+0000 7f1d700b3640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:19.034 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.040+0000 7f1d700b3640 4 set_mon_vals no callback set 2024-03-29T11:29:19.035 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.041+0000 7f1d700b3640 10 monclient: shutdown 2024-03-29T11:29:19.035 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.041+0000 7f1d700b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:19.035 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.041+0000 7f1d700b3640 1 -- 172.21.15.135:0/1982882616 >> v1:172.21.15.135:6790/0 conn(0x7f1d68149ac0 legacy=0x7f1d68149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:19.035 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.041+0000 7f1d700b3640 1 -- 172.21.15.135:0/1982882616 shutdown_connections 2024-03-29T11:29:19.035 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.041+0000 7f1d700b3640 1 -- 172.21.15.135:0/1982882616 >> 172.21.15.135:0/1982882616 conn(0x7f1d680b35e0 msgr2=0x7f1d680b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:19.038 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.046+0000 7f1d700b3640 1 -- 172.21.15.135:0/1982882616 shutdown_connections 2024-03-29T11:29:19.041 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.046+0000 7f1d700b3640 1 -- 172.21.15.135:0/1982882616 wait complete. 2024-03-29T11:29:19.041 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.047+0000 7f1d700b3640 10 monclient: build_initial_monmap 2024-03-29T11:29:19.041 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.047+0000 7f1d700b3640 10 monclient: monmap: 2024-03-29T11:29:19.041 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:29:19.041 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:19.041 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:29:19.041 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:29:19.041 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:19.041 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:19.041 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:29:19.042 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:29:19.042 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:29:19.042 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:19.042 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.047+0000 7f1d700b3640 1 librados: starting msgr at 2024-03-29T11:29:19.042 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.047+0000 7f1d700b3640 1 librados: starting objecter 2024-03-29T11:29:19.042 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.047+0000 7f1d700b3640 1 Processor -- start 2024-03-29T11:29:19.042 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.047+0000 7f1d700b3640 1 -- start start 2024-03-29T11:29:19.042 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.047+0000 7f1d700b3640 1 librados: setting wanted keys 2024-03-29T11:29:19.042 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.047+0000 7f1d700b3640 1 librados: calling monclient init 2024-03-29T11:29:19.042 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.047+0000 7f1d700b3640 10 monclient: init 2024-03-29T11:29:19.042 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.048+0000 7f1d700b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:19.042 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.048+0000 7f1d700b3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:19.042 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.048+0000 7f1d700b3640 10 monclient(hunting): picked mon.noname-c con 0x7f1d680a9750 addr v1:172.21.15.135:6790/0 2024-03-29T11:29:19.042 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.048+0000 7f1d700b3640 10 monclient(hunting): picked mon.noname-a con 0x7f1d6814fe50 addr v1:172.21.15.135:6789/0 2024-03-29T11:29:19.043 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.048+0000 7f1d700b3640 10 monclient(hunting): picked mon.noname-b con 0x7f1d68140d10 addr v1:172.21.15.194:6789/0 2024-03-29T11:29:19.043 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.048+0000 7f1d700b3640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d6806eeb0 con 0x7f1d6814fe50 2024-03-29T11:29:19.043 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.048+0000 7f1d700b3640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d680716f0 con 0x7f1d68140d10 2024-03-29T11:29:19.043 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.048+0000 7f1d700b3640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d680719d0 con 0x7f1d680a9750 2024-03-29T11:29:19.043 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.048+0000 7f1d700b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:19.043 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.048+0000 7f1d700b3640 10 monclient(hunting): authenticate will time out at 1066.416787s 2024-03-29T11:29:19.046 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.055+0000 7f1d6de4f640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f1d680a9750 0x7f1d681464f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:57871/0 (socket says 172.21.15.135:57871) 2024-03-29T11:29:19.046 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.055+0000 7f1d6de4f640 1 -- 172.21.15.135:0/3588003626 learned_addr learned my addr 172.21.15.135:0/3588003626 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:29:19.046 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.055+0000 7f1d5e7fc640 10 client.?.objecter ms_handle_connect 0x7f1d680a9750 2024-03-29T11:29:19.047 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.055+0000 7f1d5e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:19.047 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.055+0000 7f1d5e7fc640 10 client.?.objecter ms_handle_connect 0x7f1d68140d10 2024-03-29T11:29:19.047 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.055+0000 7f1d5e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:19.047 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.055+0000 7f1d5e7fc640 1 -- 172.21.15.135:0/3588003626 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3477511981 0 0) 0x7f1d680716f0 con 0x7f1d68140d10 2024-03-29T11:29:19.047 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.055+0000 7f1d5e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:19.047 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.055+0000 7f1d5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:19.047 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.055+0000 7f1d5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:19.047 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.055+0000 7f1d5e7fc640 10 monclient(hunting): my global_id is 4402 2024-03-29T11:29:19.047 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.055+0000 7f1d5e7fc640 1 -- 172.21.15.135:0/3588003626 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d4c004eb0 con 0x7f1d68140d10 2024-03-29T11:29:19.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.059+0000 7f1d5e7fc640 10 client.?.objecter ms_handle_connect 0x7f1d6814fe50 2024-03-29T11:29:19.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.059+0000 7f1d5e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:19.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.059+0000 7f1d5e7fc640 1 -- 172.21.15.135:0/3588003626 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 189805423 0 0) 0x7f1d6806eeb0 con 0x7f1d6814fe50 2024-03-29T11:29:19.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.059+0000 7f1d5e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:19.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.059+0000 7f1d5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:19.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.059+0000 7f1d5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:19.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.059+0000 7f1d5e7fc640 10 monclient(hunting): my global_id is 4404 2024-03-29T11:29:19.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.059+0000 7f1d5e7fc640 1 -- 172.21.15.135:0/3588003626 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d680716f0 con 0x7f1d6814fe50 2024-03-29T11:29:19.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.059+0000 7f1d5e7fc640 1 -- 172.21.15.135:0/3588003626 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1837849875 0 0) 0x7f1d4c004eb0 con 0x7f1d68140d10 2024-03-29T11:29:19.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.059+0000 7f1d5e7fc640 1 -- 172.21.15.135:0/3588003626 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d6806eeb0 con 0x7f1d68140d10 2024-03-29T11:29:19.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.059+0000 7f1d5e7fc640 1 -- 172.21.15.135:0/3588003626 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3142605621 0 0) 0x7f1d680719d0 con 0x7f1d680a9750 2024-03-29T11:29:19.053 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.059+0000 7f1d5e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:19.054 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.059+0000 7f1d5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:19.054 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.059+0000 7f1d5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:19.054 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.059+0000 7f1d5e7fc640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:29:19.054 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.059+0000 7f1d5e7fc640 1 -- 172.21.15.135:0/3588003626 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d4c004eb0 con 0x7f1d680a9750 2024-03-29T11:29:19.054 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.059+0000 7f1d5e7fc640 1 -- 172.21.15.135:0/3588003626 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f1d64001830 con 0x7f1d68140d10 2024-03-29T11:29:19.054 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.059+0000 7f1d5e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:19.054 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.059+0000 7f1d5e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:19.054 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.059+0000 7f1d5e7fc640 10 monclient(hunting): dump: 2024-03-29T11:29:19.054 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:19.054 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:19.054 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:19.054 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:19.054 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:19.054 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:19.055 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:19.055 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:19.055 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:19.055 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:19.055 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.059+0000 7f1d5e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:19.055 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d5e7fc640 1 -- 172.21.15.135:0/3588003626 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4083437566 0 0) 0x7f1d4c004eb0 con 0x7f1d680a9750 2024-03-29T11:29:19.055 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d5e7fc640 1 -- 172.21.15.135:0/3588003626 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d680719d0 con 0x7f1d680a9750 2024-03-29T11:29:19.055 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d5e7fc640 1 -- 172.21.15.135:0/3588003626 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f1d60001c70 con 0x7f1d680a9750 2024-03-29T11:29:19.055 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d5e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:19.055 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d5e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:19.055 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d5e7fc640 10 monclient(hunting): dump: 2024-03-29T11:29:19.055 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:19.055 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:19.055 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:19.056 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:19.056 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:19.056 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:19.056 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:19.056 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:19.056 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:19.056 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:19.056 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d5e7fc640 1 -- 172.21.15.135:0/3588003626 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1950502237 0 0) 0x7f1d6806eeb0 con 0x7f1d68140d10 2024-03-29T11:29:19.056 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d5e7fc640 1 -- 172.21.15.135:0/3588003626 >> v1:172.21.15.135:6790/0 conn(0x7f1d680a9750 legacy=0x7f1d681464f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:19.057 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d5e7fc640 1 -- 172.21.15.135:0/3588003626 >> v1:172.21.15.135:6789/0 conn(0x7f1d6814fe50 legacy=0x7f1d6806e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:19.057 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d5e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:29:19.057 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d5e7fc640 1 monclient: found mon.b 2024-03-29T11:29:19.057 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d5e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:19.057 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d5e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:29:19.057 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d5e7fc640 1 -- 172.21.15.135:0/3588003626 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d68070950 con 0x7f1d68140d10 2024-03-29T11:29:19.057 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d5e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:29:19.057 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d5e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:29:19.057 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:19.057 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d700b3640 5 monclient: authenticate success, global_id 4402 2024-03-29T11:29:19.057 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d700b3640 10 monclient: _renew_subs 2024-03-29T11:29:19.057 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d700b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:29:19.057 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d700b3640 1 -- 172.21.15.135:0/3588003626 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d680720d0 con 0x7f1d68140d10 2024-03-29T11:29:19.057 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d700b3640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:19.057 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d700b3640 10 monclient: _renew_subs 2024-03-29T11:29:19.058 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d700b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:29:19.058 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d700b3640 1 -- 172.21.15.135:0/3588003626 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d680725b0 con 0x7f1d68140d10 2024-03-29T11:29:19.058 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d700b3640 1 librados: init done 2024-03-29T11:29:19.058 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d5e7fc640 1 -- 172.21.15.135:0/3588003626 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d64001d20 con 0x7f1d68140d10 2024-03-29T11:29:19.058 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d5e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:19.058 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d5e7fc640 1 -- 172.21.15.135:0/3588003626 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f1d640037b0 con 0x7f1d68140d10 2024-03-29T11:29:19.058 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:19.058 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d5e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:19.058 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d5e7fc640 10 monclient: dump: 2024-03-29T11:29:19.058 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:19.058 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:19.059 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:19.059 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:19.059 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:19.059 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:19.059 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:19.059 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:19.059 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:19.059 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:19.059 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.060+0000 7f1d5ffff640 4 set_mon_vals no callback set 2024-03-29T11:29:19.059 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.061+0000 7f1d700b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:19.060 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.061+0000 7f1d700b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:19.060 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.061+0000 7f1d700b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:29:19.060 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.061+0000 7f1d700b3640 1 -- 172.21.15.135:0/3588003626 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d50005080 con 0x7f1d68140d10 2024-03-29T11:29:19.060 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.062+0000 7f1d5e7fc640 1 -- 172.21.15.135:0/3588003626 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f1d64003910 con 0x7f1d68140d10 2024-03-29T11:29:19.060 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.063+0000 7f1d5e7fc640 1 -- 172.21.15.135:0/3588003626 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f1d64083780 con 0x7f1d68140d10 2024-03-29T11:29:19.061 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.063+0000 7f1d5e7fc640 10 client.4402.objecter ms_dispatch 0x7f1d68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:19.061 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.063+0000 7f1d5e7fc640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:19.061 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.063+0000 7f1d5e7fc640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:19.061 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.063+0000 7f1d5e7fc640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-29T11:29:19.061 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.063+0000 7f1d6de4f640 10 monclient: get_auth_request con 0x7f1d4c06dea0 auth_method 0 2024-03-29T11:29:19.061 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.064+0000 7f1d5e7fc640 10 client.4402.objecter ms_handle_connect 0x7f1d4c06dea0 2024-03-29T11:29:19.062 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.071+0000 7f1d5e7fc640 1 -- 172.21.15.135:0/3588003626 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1d6404f060 con 0x7f1d68140d10 2024-03-29T11:29:19.062 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.071+0000 7f1d5e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:19.062 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.071+0000 7f1d5e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:19.384 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.393+0000 7f1d700b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:19.384 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.393+0000 7f1d700b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:19.384 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.393+0000 7f1d700b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:29:19.385 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.393+0000 7f1d700b3640 1 -- 172.21.15.135:0/3588003626 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d50005330 con 0x7f1d68140d10 2024-03-29T11:29:19.389 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.398+0000 7f1d5e7fc640 1 -- 172.21.15.135:0/3588003626 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f1d64050f90 con 0x7f1d68140d10 2024-03-29T11:29:19.389 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.398+0000 7f1d5e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:19.389 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.398+0000 7f1d5e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:19.392 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:29:19.393 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:19.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.404+0000 7f1d700b3640 10 librados: watch_flush enter 2024-03-29T11:29:19.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.404+0000 7f1d700b3640 10 librados: watch_flush exit 2024-03-29T11:29:19.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.404+0000 7f1d700b3640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-29T11:29:19.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.404+0000 7f1d700b3640 10 client.4402.objecter successfully canceled tick 2024-03-29T11:29:19.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.405+0000 7f1d700b3640 1 -- 172.21.15.135:0/3588003626 >> v1:172.21.15.135:6816/3469875285 conn(0x7f1d4c06dea0 legacy=0x7f1d4c070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:19.396 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.405+0000 7f1d700b3640 10 monclient: shutdown 2024-03-29T11:29:19.396 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.405+0000 7f1d700b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:19.396 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.405+0000 7f1d700b3640 1 -- 172.21.15.135:0/3588003626 >> v1:172.21.15.194:6789/0 conn(0x7f1d68140d10 legacy=0x7f1d681fffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:19.396 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.405+0000 7f1d700b3640 1 -- 172.21.15.135:0/3588003626 shutdown_connections 2024-03-29T11:29:19.396 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.405+0000 7f1d700b3640 1 -- 172.21.15.135:0/3588003626 >> 172.21.15.135:0/3588003626 conn(0x7f1d680b35e0 msgr2=0x7f1d68141710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:19.396 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.405+0000 7f1d700b3640 1 -- 172.21.15.135:0/3588003626 shutdown_connections 2024-03-29T11:29:19.397 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.406+0000 7f1d700b3640 1 -- 172.21.15.135:0/3588003626 wait complete. 2024-03-29T11:29:19.397 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:19.406+0000 7f1d700b3640 1 librados: shutdown 2024-03-29T11:29:19.415 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:26.222 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:26.375 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.384+0000 7f35c6814640 10 monclient: get_monmap_and_config 2024-03-29T11:29:26.375 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.384+0000 7f35c6814640 10 monclient: build_initial_monmap 2024-03-29T11:29:26.376 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.384+0000 7f35c6814640 10 monclient: monmap: 2024-03-29T11:29:26.376 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:29:26.376 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:26.376 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:29:26.384805+0000 2024-03-29T11:29:26.376 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:29:26.384805+0000 2024-03-29T11:29:26.376 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:26.376 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:26.376 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:29:26.376 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:29:26.376 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:29:26.376 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:26.377 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.385+0000 7f35c6814640 1 Processor -- start 2024-03-29T11:29:26.380 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.386+0000 7f35c6814640 1 -- start start 2024-03-29T11:29:26.380 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.386+0000 7f35c6814640 10 monclient: init 2024-03-29T11:29:26.380 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.386+0000 7f35c6814640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:26.380 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.386+0000 7f35c6814640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:26.380 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.386+0000 7f35c6814640 10 monclient(hunting): picked mon.noname-a con 0x7f35c00a9ac0 addr v1:172.21.15.135:6789/0 2024-03-29T11:29:26.381 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.386+0000 7f35c6814640 10 monclient(hunting): picked mon.noname-b con 0x7f35c013ee90 addr v1:172.21.15.135:6790/0 2024-03-29T11:29:26.381 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.386+0000 7f35c6814640 10 monclient(hunting): picked mon.noname-c con 0x7f35c00af900 addr v1:172.21.15.194:6789/0 2024-03-29T11:29:26.381 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.386+0000 7f35c6814640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c00afd10 con 0x7f35c00a9ac0 2024-03-29T11:29:26.381 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.386+0000 7f35c6814640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c00af6f0 con 0x7f35c00af900 2024-03-29T11:29:26.381 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.386+0000 7f35c6814640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c00aac80 con 0x7f35c013ee90 2024-03-29T11:29:26.381 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.386+0000 7f35c6814640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:26.381 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.386+0000 7f35c6814640 10 monclient(hunting): authenticate will time out at 1073.755010s 2024-03-29T11:29:26.383 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.392+0000 7f35bf7fe640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f35c013ee90 0x7f35c013b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:40319/0 (socket says 172.21.15.135:40319) 2024-03-29T11:29:26.383 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.393+0000 7f35c4db1640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f35c00af900 0x7f35c0149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:40203/0 (socket says 172.21.15.135:40203) 2024-03-29T11:29:26.384 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.393+0000 7f35c4db1640 1 -- 172.21.15.135:0/3368211742 learned_addr learned my addr 172.21.15.135:0/3368211742 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:29:26.387 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.394+0000 7f35beffd640 1 -- 172.21.15.135:0/3368211742 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672225293 0 0) 0x7f35c00af6f0 con 0x7f35c00af900 2024-03-29T11:29:26.387 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.394+0000 7f35beffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:26.387 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.394+0000 7f35beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:26.387 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.394+0000 7f35beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:26.387 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.394+0000 7f35beffd640 10 monclient(hunting): my global_id is 4405 2024-03-29T11:29:26.387 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.394+0000 7f35beffd640 1 -- 172.21.15.135:0/3368211742 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35a8003520 con 0x7f35c00af900 2024-03-29T11:29:26.388 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.394+0000 7f35beffd640 1 -- 172.21.15.135:0/3368211742 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707762952 0 0) 0x7f35c00afd10 con 0x7f35c00a9ac0 2024-03-29T11:29:26.388 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.394+0000 7f35beffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:26.388 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.394+0000 7f35beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:26.388 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.394+0000 7f35beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:26.388 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.394+0000 7f35beffd640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:29:26.388 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.394+0000 7f35beffd640 1 -- 172.21.15.135:0/3368211742 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35c00af6f0 con 0x7f35c00a9ac0 2024-03-29T11:29:26.388 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.395+0000 7f35beffd640 1 -- 172.21.15.135:0/3368211742 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2012568750 0 0) 0x7f35c00af6f0 con 0x7f35c00a9ac0 2024-03-29T11:29:26.388 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.395+0000 7f35beffd640 1 -- 172.21.15.135:0/3368211742 >> v1:172.21.15.135:6790/0 conn(0x7f35c013ee90 legacy=0x7f35c013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:26.388 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.395+0000 7f35beffd640 1 -- 172.21.15.135:0/3368211742 >> v1:172.21.15.194:6789/0 conn(0x7f35c00af900 legacy=0x7f35c0149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:26.388 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.395+0000 7f35beffd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:26.388 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.395+0000 7f35beffd640 1 monclient: found mon.noname-a 2024-03-29T11:29:26.388 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.395+0000 7f35beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:26.388 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.395+0000 7f35beffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.135:6789/0 2024-03-29T11:29:26.388 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.395+0000 7f35beffd640 1 -- 172.21.15.135:0/3368211742 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c00aaf60 con 0x7f35c00a9ac0 2024-03-29T11:29:26.388 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.395+0000 7f35beffd640 10 monclient: _finish_auth 0 2024-03-29T11:29:26.389 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.395+0000 7f35beffd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:26.389 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.395+0000 7f35beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:26.389 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.395+0000 7f35beffd640 1 -- 172.21.15.135:0/3368211742 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f35b40015a0 con 0x7f35c00a9ac0 2024-03-29T11:29:26.389 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.395+0000 7f35beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:26.389 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.395+0000 7f35beffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:26.389 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.395+0000 7f35beffd640 10 monclient: dump: 2024-03-29T11:29:26.389 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:26.389 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:26.389 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:26.389 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:26.389 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:26.389 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:26.389 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:26.389 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:26.390 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:26.390 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:26.390 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.396+0000 7f35c6814640 5 monclient: authenticate success, global_id 4407 2024-03-29T11:29:26.390 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.396+0000 7f35c6814640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:26.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.398+0000 7f35beffd640 1 -- 172.21.15.135:0/3368211742 <== mon.0 v1:172.21.15.135:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35b40017f0 con 0x7f35c00a9ac0 2024-03-29T11:29:26.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.398+0000 7f35beffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:26.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.398+0000 7f35beffd640 1 -- 172.21.15.135:0/3368211742 <== mon.0 v1:172.21.15.135:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f35b4001b80 con 0x7f35c00a9ac0 2024-03-29T11:29:26.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.398+0000 7f35beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:26.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.398+0000 7f35beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:26.394 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.398+0000 7f35beffd640 10 monclient: dump: 2024-03-29T11:29:26.394 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:26.394 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:26.394 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:26.394 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:26.395 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:26.395 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:26.395 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:26.395 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:26.395 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:26.395 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:26.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.399+0000 7f35c6814640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:26.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.399+0000 7f35c6814640 4 set_mon_vals no callback set 2024-03-29T11:29:26.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.400+0000 7f35c6814640 10 monclient: shutdown 2024-03-29T11:29:26.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.400+0000 7f35c6814640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:26.395 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.400+0000 7f35c6814640 1 -- 172.21.15.135:0/3368211742 >> v1:172.21.15.135:6789/0 conn(0x7f35c00a9ac0 legacy=0x7f35c00adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:26.396 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.404+0000 7f35c6814640 1 -- 172.21.15.135:0/3368211742 shutdown_connections 2024-03-29T11:29:26.396 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.404+0000 7f35c6814640 1 -- 172.21.15.135:0/3368211742 >> 172.21.15.135:0/3368211742 conn(0x7f35c00b35e0 msgr2=0x7f35c00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:26.396 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.405+0000 7f35c6814640 1 -- 172.21.15.135:0/3368211742 shutdown_connections 2024-03-29T11:29:26.396 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.405+0000 7f35c6814640 1 -- 172.21.15.135:0/3368211742 wait complete. 2024-03-29T11:29:26.397 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.405+0000 7f35c6814640 10 monclient: build_initial_monmap 2024-03-29T11:29:26.398 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.405+0000 7f35c6814640 10 monclient: monmap: 2024-03-29T11:29:26.398 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:29:26.398 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:26.398 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:29:26.398 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:29:26.398 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:26.398 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:26.398 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:29:26.398 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:29:26.399 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:29:26.399 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:26.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.406+0000 7f35c6814640 1 librados: starting msgr at 2024-03-29T11:29:26.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.406+0000 7f35c6814640 1 librados: starting objecter 2024-03-29T11:29:26.399 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.406+0000 7f35c6814640 1 Processor -- start 2024-03-29T11:29:26.405 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.407+0000 7f35c6814640 1 -- start start 2024-03-29T11:29:26.405 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.407+0000 7f35c6814640 1 librados: setting wanted keys 2024-03-29T11:29:26.405 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.407+0000 7f35c6814640 1 librados: calling monclient init 2024-03-29T11:29:26.405 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.407+0000 7f35c6814640 10 monclient: init 2024-03-29T11:29:26.405 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.407+0000 7f35c6814640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:26.405 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.407+0000 7f35c6814640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:26.405 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.407+0000 7f35c6814640 10 monclient(hunting): picked mon.noname-a con 0x7f35c00a9ac0 addr v1:172.21.15.135:6789/0 2024-03-29T11:29:26.405 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.407+0000 7f35c6814640 10 monclient(hunting): picked mon.noname-b con 0x7f35c00af900 addr v1:172.21.15.194:6789/0 2024-03-29T11:29:26.405 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.407+0000 7f35c6814640 10 monclient(hunting): picked mon.noname-c con 0x7f35c013ee90 addr v1:172.21.15.135:6790/0 2024-03-29T11:29:26.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.407+0000 7f35c6814640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c01c43d0 con 0x7f35c00a9ac0 2024-03-29T11:29:26.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.407+0000 7f35c6814640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c01c6230 con 0x7f35c00af900 2024-03-29T11:29:26.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.407+0000 7f35c6814640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c01c6510 con 0x7f35c013ee90 2024-03-29T11:29:26.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.407+0000 7f35c6814640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:26.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.412+0000 7f35c4db1640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f35c013ee90 0x7f35c0137b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:40461/0 (socket says 172.21.15.135:40461) 2024-03-29T11:29:26.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.412+0000 7f35c4db1640 1 -- 172.21.15.135:0/2308973144 learned_addr learned my addr 172.21.15.135:0/2308973144 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:29:26.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.407+0000 7f35c6814640 10 monclient(hunting): authenticate will time out at 1073.775960s 2024-03-29T11:29:26.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.412+0000 7f35bcff9640 10 client.?.objecter ms_handle_connect 0x7f35c013ee90 2024-03-29T11:29:26.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.412+0000 7f35bcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:26.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.413+0000 7f35bcff9640 1 -- 172.21.15.135:0/2308973144 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1534006445 0 0) 0x7f35c01c6510 con 0x7f35c013ee90 2024-03-29T11:29:26.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.413+0000 7f35bcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:26.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.413+0000 7f35bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:26.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.413+0000 7f35bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:26.406 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.413+0000 7f35bcff9640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:29:26.407 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.413+0000 7f35bcff9640 1 -- 172.21.15.135:0/2308973144 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35ac004eb0 con 0x7f35c013ee90 2024-03-29T11:29:26.411 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.416+0000 7f35bcff9640 10 client.?.objecter ms_handle_connect 0x7f35c00af900 2024-03-29T11:29:26.411 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.416+0000 7f35bcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:26.411 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.416+0000 7f35bcff9640 1 -- 172.21.15.135:0/2308973144 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3981236466 0 0) 0x7f35ac004eb0 con 0x7f35c013ee90 2024-03-29T11:29:26.412 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.416+0000 7f35bcff9640 1 -- 172.21.15.135:0/2308973144 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35ac005300 con 0x7f35c013ee90 2024-03-29T11:29:26.412 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.416+0000 7f35bcff9640 1 -- 172.21.15.135:0/2308973144 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f35b8001c00 con 0x7f35c013ee90 2024-03-29T11:29:26.412 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.416+0000 7f35bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:26.412 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.416+0000 7f35bcff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:26.412 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.416+0000 7f35bcff9640 10 monclient(hunting): dump: 2024-03-29T11:29:26.412 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:26.412 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:26.412 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:26.412 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:26.412 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:26.412 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:26.412 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:26.412 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:26.412 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:26.412 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:26.413 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.416+0000 7f35bcff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:26.413 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.416+0000 7f35bcff9640 1 -- 172.21.15.135:0/2308973144 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608843114 0 0) 0x7f35c01c6230 con 0x7f35c00af900 2024-03-29T11:29:26.413 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.416+0000 7f35bcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:26.413 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.416+0000 7f35bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:26.413 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.416+0000 7f35bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:26.413 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.416+0000 7f35bcff9640 10 monclient(hunting): my global_id is 4408 2024-03-29T11:29:26.413 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.416+0000 7f35bcff9640 1 -- 172.21.15.135:0/2308973144 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35ac004eb0 con 0x7f35c00af900 2024-03-29T11:29:26.413 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.417+0000 7f35bcff9640 1 -- 172.21.15.135:0/2308973144 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3887344437 0 0) 0x7f35ac005300 con 0x7f35c013ee90 2024-03-29T11:29:26.413 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.417+0000 7f35bcff9640 1 -- 172.21.15.135:0/2308973144 >> v1:172.21.15.194:6789/0 conn(0x7f35c00af900 legacy=0x7f35c006fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:26.413 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.417+0000 7f35bcff9640 1 -- 172.21.15.135:0/2308973144 >> v1:172.21.15.135:6789/0 conn(0x7f35c00a9ac0 legacy=0x7f35c01dee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:26.413 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.417+0000 7f35bcff9640 10 monclient: _finish_hunting 0 2024-03-29T11:29:26.413 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.417+0000 7f35bcff9640 1 monclient: found mon.c 2024-03-29T11:29:26.413 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.417+0000 7f35bcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:26.413 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.417+0000 7f35bcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:29:26.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.417+0000 7f35bcff9640 1 -- 172.21.15.135:0/2308973144 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c0072600 con 0x7f35c013ee90 2024-03-29T11:29:26.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.417+0000 7f35bcff9640 10 monclient: _finish_auth 0 2024-03-29T11:29:26.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.417+0000 7f35bcff9640 10 monclient: _check_auth_tickets 2024-03-29T11:29:26.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.417+0000 7f35bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:26.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.417+0000 7f35c6814640 5 monclient: authenticate success, global_id 4412 2024-03-29T11:29:26.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.417+0000 7f35c6814640 10 monclient: _renew_subs 2024-03-29T11:29:26.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.417+0000 7f35c6814640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:29:26.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.417+0000 7f35c6814640 1 -- 172.21.15.135:0/2308973144 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c01c4620 con 0x7f35c013ee90 2024-03-29T11:29:26.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.417+0000 7f35c6814640 10 client.4412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:26.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.417+0000 7f35c6814640 10 monclient: _renew_subs 2024-03-29T11:29:26.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.417+0000 7f35c6814640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:29:26.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.417+0000 7f35c6814640 1 -- 172.21.15.135:0/2308973144 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c01c4a90 con 0x7f35c013ee90 2024-03-29T11:29:26.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.417+0000 7f35c6814640 1 librados: init done 2024-03-29T11:29:26.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.419+0000 7f35bcff9640 1 -- 172.21.15.135:0/2308973144 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35b8002070 con 0x7f35c013ee90 2024-03-29T11:29:26.414 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.419+0000 7f35bcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:26.415 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.419+0000 7f35bcff9640 1 -- 172.21.15.135:0/2308973144 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f35b8004d00 con 0x7f35c013ee90 2024-03-29T11:29:26.415 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.419+0000 7f35bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:26.415 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.419+0000 7f35bcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:26.415 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.419+0000 7f35bcff9640 10 monclient: dump: 2024-03-29T11:29:26.415 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:26.415 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:26.415 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:26.415 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:26.415 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:26.415 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:26.415 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:26.415 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:26.415 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:26.415 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:26.416 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.419+0000 7f35bdffb640 4 set_mon_vals no callback set 2024-03-29T11:29:26.416 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.419+0000 7f35c6814640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:26.416 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.421+0000 7f35c6814640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:26.416 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.421+0000 7f35c6814640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:29:26.416 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.421+0000 7f35c6814640 1 -- 172.21.15.135:0/2308973144 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a0005080 con 0x7f35c013ee90 2024-03-29T11:29:26.416 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.421+0000 7f35bcff9640 1 -- 172.21.15.135:0/2308973144 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f35b8004e60 con 0x7f35c013ee90 2024-03-29T11:29:26.416 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.422+0000 7f35bcff9640 1 -- 172.21.15.135:0/2308973144 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f35b8065b00 con 0x7f35c013ee90 2024-03-29T11:29:26.417 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.422+0000 7f35bcff9640 10 client.4412.objecter ms_dispatch 0x7f35c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:26.417 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.422+0000 7f35bcff9640 3 client.4412.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:26.417 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.422+0000 7f35bcff9640 3 client.4412.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:26.417 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.422+0000 7f35bcff9640 20 client.4412.objecter dump_active .. 0 homeless 2024-03-29T11:29:26.417 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.423+0000 7f35bffff640 10 monclient: get_auth_request con 0x7f35ac06dec0 auth_method 0 2024-03-29T11:29:26.418 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.428+0000 7f35bcff9640 10 client.4412.objecter ms_handle_connect 0x7f35ac06dec0 2024-03-29T11:29:26.422 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.431+0000 7f35bcff9640 1 -- 172.21.15.135:0/2308973144 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f35b8061da0 con 0x7f35c013ee90 2024-03-29T11:29:26.422 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.431+0000 7f35bcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:26.422 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.431+0000 7f35bcff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:26.749 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.758+0000 7f35c6814640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:26.749 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.758+0000 7f35c6814640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:26.749 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.758+0000 7f35c6814640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:29:26.749 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.758+0000 7f35c6814640 1 -- 172.21.15.135:0/2308973144 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35a0005330 con 0x7f35c013ee90 2024-03-29T11:29:26.750 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.760+0000 7f35bcff9640 1 -- 172.21.15.135:0/2308973144 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f35b8061f40 con 0x7f35c013ee90 2024-03-29T11:29:26.751 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:29:26.751 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.760+0000 7f35bcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:26.751 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.760+0000 7f35bcff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:26.752 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:26.754 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.764+0000 7f35c6814640 10 librados: watch_flush enter 2024-03-29T11:29:26.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.764+0000 7f35c6814640 10 librados: watch_flush exit 2024-03-29T11:29:26.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.764+0000 7f35c6814640 20 client.4412.objecter shutdown clearing up homeless session... 2024-03-29T11:29:26.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.764+0000 7f35c6814640 10 client.4412.objecter successfully canceled tick 2024-03-29T11:29:26.756 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.765+0000 7f35c6814640 1 -- 172.21.15.135:0/2308973144 >> v1:172.21.15.135:6816/3469875285 conn(0x7f35ac06dec0 legacy=0x7f35ac070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:26.756 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.765+0000 7f35c6814640 10 monclient: shutdown 2024-03-29T11:29:26.756 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.765+0000 7f35c6814640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:26.756 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.765+0000 7f35c6814640 1 -- 172.21.15.135:0/2308973144 >> v1:172.21.15.135:6790/0 conn(0x7f35c013ee90 legacy=0x7f35c0137b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:26.759 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.768+0000 7f35c6814640 1 -- 172.21.15.135:0/2308973144 shutdown_connections 2024-03-29T11:29:26.759 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.768+0000 7f35c6814640 1 -- 172.21.15.135:0/2308973144 >> 172.21.15.135:0/2308973144 conn(0x7f35c00b35e0 msgr2=0x7f35c013af70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:26.759 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.768+0000 7f35c6814640 1 -- 172.21.15.135:0/2308973144 shutdown_connections 2024-03-29T11:29:26.759 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.769+0000 7f35c6814640 1 -- 172.21.15.135:0/2308973144 wait complete. 2024-03-29T11:29:26.759 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:26.769+0000 7f35c6814640 1 librados: shutdown 2024-03-29T11:29:26.780 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:33.587 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:33.739 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.748+0000 7f13ad863640 10 monclient: get_monmap_and_config 2024-03-29T11:29:33.740 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.748+0000 7f13ad863640 10 monclient: build_initial_monmap 2024-03-29T11:29:33.740 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.748+0000 7f13ad863640 10 monclient: monmap: 2024-03-29T11:29:33.740 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:29:33.740 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:33.740 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:29:33.749409+0000 2024-03-29T11:29:33.740 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:29:33.749409+0000 2024-03-29T11:29:33.740 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:33.740 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:33.740 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:29:33.740 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:29:33.740 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:29:33.740 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:33.742 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.751+0000 7f13ad863640 1 Processor -- start 2024-03-29T11:29:33.742 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.751+0000 7f13ad863640 1 -- start start 2024-03-29T11:29:33.742 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.751+0000 7f13ad863640 10 monclient: init 2024-03-29T11:29:33.743 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.752+0000 7f13ad863640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:33.743 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.752+0000 7f13ad863640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:33.743 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.752+0000 7f13ad863640 10 monclient(hunting): picked mon.noname-b con 0x7f13a80b1f60 addr v1:172.21.15.135:6790/0 2024-03-29T11:29:33.743 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.752+0000 7f13ad863640 10 monclient(hunting): picked mon.noname-a con 0x7f13a8071eb0 addr v1:172.21.15.135:6789/0 2024-03-29T11:29:33.743 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.752+0000 7f13ad863640 10 monclient(hunting): picked mon.noname-c con 0x7f13a8143d10 addr v1:172.21.15.194:6789/0 2024-03-29T11:29:33.743 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.752+0000 7f13ad863640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a806ef80 con 0x7f13a8071eb0 2024-03-29T11:29:33.743 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.752+0000 7f13ad863640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a806f260 con 0x7f13a8143d10 2024-03-29T11:29:33.743 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.752+0000 7f13ad863640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a80b40c0 con 0x7f13a80b1f60 2024-03-29T11:29:33.743 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.752+0000 7f13ad863640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:33.743 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.752+0000 7f13ad863640 10 monclient(hunting): authenticate will time out at 1081.120165s 2024-03-29T11:29:33.752 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.756+0000 7f13a6ffd640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f13a80b1f60 0x7f13a8143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:46693/0 (socket says 172.21.15.135:46693) 2024-03-29T11:29:33.752 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.756+0000 7f13a6ffd640 1 -- 172.21.15.135:0/3314519805 learned_addr learned my addr 172.21.15.135:0/3314519805 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:29:33.752 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.757+0000 7f13a5ffb640 1 -- 172.21.15.135:0/3314519805 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4131549913 0 0) 0x7f13a80b40c0 con 0x7f13a80b1f60 2024-03-29T11:29:33.752 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.757+0000 7f13a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:33.752 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.757+0000 7f13a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:33.752 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.757+0000 7f13a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:33.752 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.757+0000 7f13a5ffb640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:29:33.753 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.757+0000 7f13a5ffb640 1 -- 172.21.15.135:0/3314519805 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1390003520 con 0x7f13a80b1f60 2024-03-29T11:29:33.753 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.758+0000 7f13a5ffb640 1 -- 172.21.15.135:0/3314519805 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3525237173 0 0) 0x7f1390003520 con 0x7f13a80b1f60 2024-03-29T11:29:33.753 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.758+0000 7f13a5ffb640 1 -- 172.21.15.135:0/3314519805 >> v1:172.21.15.194:6789/0 conn(0x7f13a8143d10 legacy=0x7f13a8146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:33.753 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.758+0000 7f13a5ffb640 1 -- 172.21.15.135:0/3314519805 >> v1:172.21.15.135:6789/0 conn(0x7f13a8071eb0 legacy=0x7f13a8146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:33.753 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.758+0000 7f13a5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:29:33.753 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.758+0000 7f13a5ffb640 1 monclient: found mon.noname-b 2024-03-29T11:29:33.753 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.758+0000 7f13a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:33.753 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.758+0000 7f13a5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:29:33.753 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.758+0000 7f13a5ffb640 1 -- 172.21.15.135:0/3314519805 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a8155d70 con 0x7f13a80b1f60 2024-03-29T11:29:33.753 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.758+0000 7f13a5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:29:33.753 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.758+0000 7f13a5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:29:33.753 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.758+0000 7f13a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:33.753 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.758+0000 7f13a5ffb640 1 -- 172.21.15.135:0/3314519805 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f139c001690 con 0x7f13a80b1f60 2024-03-29T11:29:33.753 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.759+0000 7f13a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:33.754 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.759+0000 7f13a5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:33.754 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.759+0000 7f13a5ffb640 10 monclient: dump: 2024-03-29T11:29:33.754 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:33.754 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:33.754 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:33.754 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:33.754 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:33.754 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:33.754 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:33.754 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:33.754 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:33.754 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:33.754 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.759+0000 7f13ad863640 5 monclient: authenticate success, global_id 4415 2024-03-29T11:29:33.754 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.759+0000 7f13ad863640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:33.754 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.759+0000 7f13a5ffb640 1 -- 172.21.15.135:0/3314519805 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f139c001920 con 0x7f13a80b1f60 2024-03-29T11:29:33.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.759+0000 7f13a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:33.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.759+0000 7f13a5ffb640 1 -- 172.21.15.135:0/3314519805 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f139c001cb0 con 0x7f13a80b1f60 2024-03-29T11:29:33.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.759+0000 7f13ad863640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:33.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.759+0000 7f13ad863640 4 set_mon_vals no callback set 2024-03-29T11:29:33.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.759+0000 7f13a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:33.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.759+0000 7f13a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:33.755 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.759+0000 7f13a5ffb640 10 monclient: dump: 2024-03-29T11:29:33.755 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:33.755 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:33.755 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:33.755 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:33.755 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:33.755 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:33.755 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:33.755 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:33.756 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:33.756 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:33.756 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.760+0000 7f13ad863640 10 monclient: shutdown 2024-03-29T11:29:33.756 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.760+0000 7f13ad863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:33.756 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.760+0000 7f13ad863640 1 -- 172.21.15.135:0/3314519805 >> v1:172.21.15.135:6790/0 conn(0x7f13a80b1f60 legacy=0x7f13a8143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:33.756 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.760+0000 7f13ad863640 1 -- 172.21.15.135:0/3314519805 shutdown_connections 2024-03-29T11:29:33.756 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.760+0000 7f13ad863640 1 -- 172.21.15.135:0/3314519805 >> 172.21.15.135:0/3314519805 conn(0x7f13a8066a00 msgr2=0x7f13a8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:33.757 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.763+0000 7f13ad863640 1 -- 172.21.15.135:0/3314519805 shutdown_connections 2024-03-29T11:29:33.757 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.764+0000 7f13ad863640 1 -- 172.21.15.135:0/3314519805 wait complete. 2024-03-29T11:29:33.757 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.765+0000 7f13ad863640 10 monclient: build_initial_monmap 2024-03-29T11:29:33.757 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.765+0000 7f13ad863640 10 monclient: monmap: 2024-03-29T11:29:33.757 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:29:33.757 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:33.757 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:29:33.757 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:29:33.758 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:33.758 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:33.758 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:29:33.758 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:29:33.758 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:29:33.758 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:33.758 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.765+0000 7f13ad863640 1 librados: starting msgr at 2024-03-29T11:29:33.758 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.765+0000 7f13ad863640 1 librados: starting objecter 2024-03-29T11:29:33.758 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.765+0000 7f13ad863640 1 Processor -- start 2024-03-29T11:29:33.759 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.767+0000 7f13ad863640 1 -- start start 2024-03-29T11:29:33.759 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.767+0000 7f13ad863640 1 librados: setting wanted keys 2024-03-29T11:29:33.759 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.767+0000 7f13ad863640 1 librados: calling monclient init 2024-03-29T11:29:33.759 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.767+0000 7f13ad863640 10 monclient: init 2024-03-29T11:29:33.759 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.768+0000 7f13ad863640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:33.759 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.768+0000 7f13ad863640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:33.759 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.768+0000 7f13ad863640 10 monclient(hunting): picked mon.noname-c con 0x7f13a8143d10 addr v1:172.21.15.135:6790/0 2024-03-29T11:29:33.759 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.768+0000 7f13ad863640 10 monclient(hunting): picked mon.noname-b con 0x7f13a8136d70 addr v1:172.21.15.194:6789/0 2024-03-29T11:29:33.759 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.768+0000 7f13ad863640 10 monclient(hunting): picked mon.noname-a con 0x7f13a81df3d0 addr v1:172.21.15.135:6789/0 2024-03-29T11:29:33.759 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.768+0000 7f13ad863640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a8142e40 con 0x7f13a81df3d0 2024-03-29T11:29:33.759 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.768+0000 7f13ad863640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a8143070 con 0x7f13a8136d70 2024-03-29T11:29:33.759 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.768+0000 7f13ad863640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a8132740 con 0x7f13a8143d10 2024-03-29T11:29:33.759 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.768+0000 7f13ad863640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:33.759 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.768+0000 7f13ad863640 10 monclient(hunting): authenticate will time out at 1081.136165s 2024-03-29T11:29:33.768 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.772+0000 7f13a77fe640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f13a81df3d0 0x7f13a8132030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:32919/0 (socket says 172.21.15.135:32919) 2024-03-29T11:29:33.768 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.772+0000 7f13a77fe640 1 -- 172.21.15.135:0/4007769857 learned_addr learned my addr 172.21.15.135:0/4007769857 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:29:33.768 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.772+0000 7f138ffff640 10 client.?.objecter ms_handle_connect 0x7f13a81df3d0 2024-03-29T11:29:33.768 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.772+0000 7f138ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:33.769 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.772+0000 7f138ffff640 10 client.?.objecter ms_handle_connect 0x7f13a8136d70 2024-03-29T11:29:33.769 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.772+0000 7f138ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:33.769 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.773+0000 7f138ffff640 1 -- 172.21.15.135:0/4007769857 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065989715 0 0) 0x7f13a8143070 con 0x7f13a8136d70 2024-03-29T11:29:33.769 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.773+0000 7f138ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:33.769 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.773+0000 7f138ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:33.769 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.773+0000 7f138ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:33.769 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.773+0000 7f138ffff640 10 monclient(hunting): my global_id is 4411 2024-03-29T11:29:33.769 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.773+0000 7f138ffff640 1 -- 172.21.15.135:0/4007769857 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1394004eb0 con 0x7f13a8136d70 2024-03-29T11:29:33.769 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.773+0000 7f138ffff640 1 -- 172.21.15.135:0/4007769857 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2817442944 0 0) 0x7f13a8142e40 con 0x7f13a81df3d0 2024-03-29T11:29:33.769 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.773+0000 7f138ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:33.770 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.773+0000 7f138ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:33.770 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.773+0000 7f138ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:33.770 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.773+0000 7f138ffff640 10 monclient(hunting): my global_id is 4410 2024-03-29T11:29:33.770 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.773+0000 7f138ffff640 1 -- 172.21.15.135:0/4007769857 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13a8143070 con 0x7f13a81df3d0 2024-03-29T11:29:33.770 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f138ffff640 1 -- 172.21.15.135:0/4007769857 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2433055530 0 0) 0x7f13a8143070 con 0x7f13a81df3d0 2024-03-29T11:29:33.770 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f138ffff640 1 -- 172.21.15.135:0/4007769857 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13a8142e40 con 0x7f13a81df3d0 2024-03-29T11:29:33.771 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f138ffff640 1 -- 172.21.15.135:0/4007769857 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f13a0001a90 con 0x7f13a81df3d0 2024-03-29T11:29:33.771 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f138ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:33.771 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f138ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:33.771 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f138ffff640 10 monclient(hunting): dump: 2024-03-29T11:29:33.771 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:33.771 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:33.771 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:33.771 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:33.771 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:33.771 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:33.771 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:33.771 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:33.771 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:33.771 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:33.772 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f138ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:33.772 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f138ffff640 1 -- 172.21.15.135:0/4007769857 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4012822421 0 0) 0x7f1394004eb0 con 0x7f13a8136d70 2024-03-29T11:29:33.772 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f138ffff640 1 -- 172.21.15.135:0/4007769857 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13a8143070 con 0x7f13a8136d70 2024-03-29T11:29:33.772 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f138ffff640 1 -- 172.21.15.135:0/4007769857 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f13980017f0 con 0x7f13a8136d70 2024-03-29T11:29:33.772 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f138ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:33.772 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f138ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:33.772 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f138ffff640 10 monclient(hunting): dump: 2024-03-29T11:29:33.772 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:33.772 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:33.773 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:33.773 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:33.773 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:33.773 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:33.773 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:33.773 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:33.773 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:33.773 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:33.773 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f138ffff640 1 -- 172.21.15.135:0/4007769857 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2838850375 0 0) 0x7f13a8143070 con 0x7f13a8136d70 2024-03-29T11:29:33.773 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f138ffff640 1 -- 172.21.15.135:0/4007769857 >> v1:172.21.15.135:6790/0 conn(0x7f13a8143d10 legacy=0x7f13a81decc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:33.773 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f138ffff640 1 -- 172.21.15.135:0/4007769857 >> v1:172.21.15.135:6789/0 conn(0x7f13a81df3d0 legacy=0x7f13a8132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:33.773 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f138ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:29:33.773 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f138ffff640 1 monclient: found mon.b 2024-03-29T11:29:33.773 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f138ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:33.773 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f138ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:29:33.774 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f138ffff640 1 -- 172.21.15.135:0/4007769857 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a81423f0 con 0x7f13a8136d70 2024-03-29T11:29:33.774 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f138ffff640 10 monclient: _finish_auth 0 2024-03-29T11:29:33.774 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f138ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:29:33.774 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f138ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:33.774 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f13ad863640 5 monclient: authenticate success, global_id 4411 2024-03-29T11:29:33.774 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f13ad863640 10 monclient: _renew_subs 2024-03-29T11:29:33.774 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f13ad863640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:29:33.774 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.774+0000 7f13ad863640 1 -- 172.21.15.135:0/4007769857 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a81426d0 con 0x7f13a8136d70 2024-03-29T11:29:33.774 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.775+0000 7f13ad863640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:33.774 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.775+0000 7f13ad863640 10 monclient: _renew_subs 2024-03-29T11:29:33.774 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.775+0000 7f13ad863640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:29:33.774 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.775+0000 7f138ffff640 1 -- 172.21.15.135:0/4007769857 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1398001d00 con 0x7f13a8136d70 2024-03-29T11:29:33.775 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.775+0000 7f13ad863640 1 -- 172.21.15.135:0/4007769857 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a8141090 con 0x7f13a8136d70 2024-03-29T11:29:33.775 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.775+0000 7f13ad863640 1 librados: init done 2024-03-29T11:29:33.775 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.775+0000 7f138ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:33.775 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.775+0000 7f138ffff640 1 -- 172.21.15.135:0/4007769857 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f1398003790 con 0x7f13a8136d70 2024-03-29T11:29:33.775 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.775+0000 7f138ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:33.775 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.775+0000 7f138ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:33.776 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.775+0000 7f138ffff640 10 monclient: dump: 2024-03-29T11:29:33.776 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:33.776 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:33.776 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:33.776 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:33.776 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:33.776 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:33.776 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:33.776 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:33.776 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:33.777 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:33.777 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.775+0000 7f13a57fa640 4 set_mon_vals no callback set 2024-03-29T11:29:33.777 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.776+0000 7f13ad863640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:33.777 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.776+0000 7f13ad863640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:33.777 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.776+0000 7f13ad863640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:29:33.777 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.776+0000 7f13ad863640 1 -- 172.21.15.135:0/4007769857 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1388005080 con 0x7f13a8136d70 2024-03-29T11:29:33.778 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.778+0000 7f138ffff640 1 -- 172.21.15.135:0/4007769857 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f13980197e0 con 0x7f13a8136d70 2024-03-29T11:29:33.778 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.779+0000 7f13a6ffd640 10 monclient: get_auth_request con 0x7f139406dfd0 auth_method 0 2024-03-29T11:29:33.779 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.779+0000 7f138ffff640 1 -- 172.21.15.135:0/4007769857 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f13980533b0 con 0x7f13a8136d70 2024-03-29T11:29:33.779 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.779+0000 7f138ffff640 10 client.4411.objecter ms_dispatch 0x7f13a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:33.779 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.779+0000 7f138ffff640 3 client.4411.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:33.779 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.779+0000 7f138ffff640 3 client.4411.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:33.779 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.779+0000 7f138ffff640 20 client.4411.objecter dump_active .. 0 homeless 2024-03-29T11:29:33.779 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.780+0000 7f138ffff640 10 client.4411.objecter ms_handle_connect 0x7f139406dfd0 2024-03-29T11:29:33.779 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.785+0000 7f138ffff640 1 -- 172.21.15.135:0/4007769857 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f139804f650 con 0x7f13a8136d70 2024-03-29T11:29:33.779 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.785+0000 7f138ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:33.779 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:33.785+0000 7f138ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:34.100 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:34.109+0000 7f13ad863640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:34.100 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:34.109+0000 7f13ad863640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:34.100 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:34.109+0000 7f13ad863640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:29:34.100 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:34.109+0000 7f13ad863640 1 -- 172.21.15.135:0/4007769857 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1388005330 con 0x7f13a8136d70 2024-03-29T11:29:34.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:34.111+0000 7f138ffff640 1 -- 172.21.15.135:0/4007769857 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f139804f650 con 0x7f13a8136d70 2024-03-29T11:29:34.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:34.111+0000 7f138ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:34.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:34.111+0000 7f138ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:34.102 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:29:34.102 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:34.106 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:34.114+0000 7f13ad863640 10 librados: watch_flush enter 2024-03-29T11:29:34.106 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:34.114+0000 7f13ad863640 10 librados: watch_flush exit 2024-03-29T11:29:34.106 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:34.114+0000 7f13ad863640 20 client.4411.objecter shutdown clearing up homeless session... 2024-03-29T11:29:34.106 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:34.114+0000 7f13ad863640 10 client.4411.objecter successfully canceled tick 2024-03-29T11:29:34.107 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:34.116+0000 7f13ad863640 1 -- 172.21.15.135:0/4007769857 >> v1:172.21.15.135:6816/3469875285 conn(0x7f139406dfd0 legacy=0x7f1394070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:34.107 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:34.116+0000 7f13ad863640 10 monclient: shutdown 2024-03-29T11:29:34.107 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:34.116+0000 7f13ad863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:34.107 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:34.116+0000 7f13ad863640 1 -- 172.21.15.135:0/4007769857 >> v1:172.21.15.194:6789/0 conn(0x7f13a8136d70 legacy=0x7f13a8131920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:34.107 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:34.116+0000 7f13ad863640 1 -- 172.21.15.135:0/4007769857 shutdown_connections 2024-03-29T11:29:34.107 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:34.116+0000 7f13ad863640 1 -- 172.21.15.135:0/4007769857 >> 172.21.15.135:0/4007769857 conn(0x7f13a8066a00 msgr2=0x7f13a8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:34.107 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:34.116+0000 7f13ad863640 1 -- 172.21.15.135:0/4007769857 shutdown_connections 2024-03-29T11:29:34.108 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:34.117+0000 7f13ad863640 1 -- 172.21.15.135:0/4007769857 wait complete. 2024-03-29T11:29:34.108 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:34.117+0000 7f13ad863640 1 librados: shutdown 2024-03-29T11:29:34.128 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:40.936 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:41.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.109+0000 7fca70225640 10 monclient: get_monmap_and_config 2024-03-29T11:29:41.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.109+0000 7fca70225640 10 monclient: build_initial_monmap 2024-03-29T11:29:41.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.109+0000 7fca70225640 10 monclient: monmap: 2024-03-29T11:29:41.102 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:29:41.102 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:41.102 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:29:41.111198+0000 2024-03-29T11:29:41.102 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:29:41.111198+0000 2024-03-29T11:29:41.102 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:41.102 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:41.103 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:29:41.103 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:29:41.103 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:29:41.103 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:41.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.110+0000 7fca70225640 1 Processor -- start 2024-03-29T11:29:41.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.111+0000 7fca70225640 1 -- start start 2024-03-29T11:29:41.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.111+0000 7fca70225640 10 monclient: init 2024-03-29T11:29:41.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.112+0000 7fca70225640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:41.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.112+0000 7fca70225640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:41.105 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.112+0000 7fca70225640 10 monclient(hunting): picked mon.noname-a con 0x7fca680b0860 addr v1:172.21.15.135:6789/0 2024-03-29T11:29:41.105 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.112+0000 7fca70225640 10 monclient(hunting): picked mon.noname-c con 0x7fca68072450 addr v1:172.21.15.194:6789/0 2024-03-29T11:29:41.105 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.112+0000 7fca70225640 10 monclient(hunting): picked mon.noname-b con 0x7fca6813ee40 addr v1:172.21.15.135:6790/0 2024-03-29T11:29:41.105 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.112+0000 7fca70225640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca680ad840 con 0x7fca680b0860 2024-03-29T11:29:41.105 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.112+0000 7fca70225640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca6813c830 con 0x7fca68072450 2024-03-29T11:29:41.105 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.112+0000 7fca70225640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca6813c9c0 con 0x7fca6813ee40 2024-03-29T11:29:41.106 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.112+0000 7fca70225640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:41.106 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.112+0000 7fca70225640 10 monclient(hunting): authenticate will time out at 1088.481207s 2024-03-29T11:29:41.106 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.112+0000 7fca6d7c0640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fca68072450 0x7fca6813c120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:60438/0 (socket says 172.21.15.135:60438) 2024-03-29T11:29:41.106 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.112+0000 7fca6d7c0640 1 -- 172.21.15.135:0/2298667877 learned_addr learned my addr 172.21.15.135:0/2298667877 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:29:41.106 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.113+0000 7fca6cfbf640 1 -- 172.21.15.135:0/2298667877 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2416642108 0 0) 0x7fca6813c830 con 0x7fca68072450 2024-03-29T11:29:41.106 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.113+0000 7fca6cfbf640 10 monclient(hunting): _negotiate 2024-03-29T11:29:41.106 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.113+0000 7fca6cfbf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:41.106 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.113+0000 7fca6cfbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:41.106 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.113+0000 7fca6cfbf640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:29:41.106 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.113+0000 7fca6cfbf640 1 -- 172.21.15.135:0/2298667877 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca54003520 con 0x7fca68072450 2024-03-29T11:29:41.108 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 1 -- 172.21.15.135:0/2298667877 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3719359805 0 0) 0x7fca6813c9c0 con 0x7fca6813ee40 2024-03-29T11:29:41.108 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 10 monclient(hunting): _negotiate 2024-03-29T11:29:41.108 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:41.108 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:41.108 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:29:41.108 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 1 -- 172.21.15.135:0/2298667877 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca6813c830 con 0x7fca6813ee40 2024-03-29T11:29:41.108 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 1 -- 172.21.15.135:0/2298667877 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866467305 0 0) 0x7fca680ad840 con 0x7fca680b0860 2024-03-29T11:29:41.108 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 10 monclient(hunting): _negotiate 2024-03-29T11:29:41.108 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:41.108 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:41.108 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 10 monclient(hunting): my global_id is 4413 2024-03-29T11:29:41.108 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 1 -- 172.21.15.135:0/2298667877 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca6813c9c0 con 0x7fca680b0860 2024-03-29T11:29:41.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 1 -- 172.21.15.135:0/2298667877 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 562427832 0 0) 0x7fca54003520 con 0x7fca68072450 2024-03-29T11:29:41.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 1 -- 172.21.15.135:0/2298667877 >> v1:172.21.15.135:6790/0 conn(0x7fca6813ee40 legacy=0x7fca6813f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:41.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 1 -- 172.21.15.135:0/2298667877 >> v1:172.21.15.135:6789/0 conn(0x7fca680b0860 legacy=0x7fca680acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:41.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 10 monclient: _finish_hunting 0 2024-03-29T11:29:41.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 1 monclient: found mon.noname-c 2024-03-29T11:29:41.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:41.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:29:41.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 1 -- 172.21.15.135:0/2298667877 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca6813f970 con 0x7fca68072450 2024-03-29T11:29:41.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 10 monclient: _finish_auth 0 2024-03-29T11:29:41.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 10 monclient: _check_auth_tickets 2024-03-29T11:29:41.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:41.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 1 -- 172.21.15.135:0/2298667877 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fca60001a10 con 0x7fca68072450 2024-03-29T11:29:41.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:41.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:41.110 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca6cfbf640 10 monclient: dump: 2024-03-29T11:29:41.110 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:41.110 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:41.110 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:41.110 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:41.110 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:41.110 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:41.110 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:41.110 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:41.111 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:41.111 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:41.111 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca70225640 5 monclient: authenticate success, global_id 4414 2024-03-29T11:29:41.111 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.114+0000 7fca70225640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:41.111 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.115+0000 7fca6cfbf640 1 -- 172.21.15.135:0/2298667877 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca60001b70 con 0x7fca68072450 2024-03-29T11:29:41.111 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.115+0000 7fca6cfbf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:41.111 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.115+0000 7fca6cfbf640 1 -- 172.21.15.135:0/2298667877 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fca60001f80 con 0x7fca68072450 2024-03-29T11:29:41.111 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.115+0000 7fca6cfbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:41.111 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.115+0000 7fca6cfbf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:41.111 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.115+0000 7fca6cfbf640 10 monclient: dump: 2024-03-29T11:29:41.111 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:41.111 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:41.111 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:41.112 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:41.112 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:41.112 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:41.112 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:41.112 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:41.112 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:41.112 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:41.112 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.115+0000 7fca70225640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:41.112 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.115+0000 7fca70225640 4 set_mon_vals no callback set 2024-03-29T11:29:41.115 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.116+0000 7fca70225640 10 monclient: shutdown 2024-03-29T11:29:41.115 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.116+0000 7fca70225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:41.116 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.116+0000 7fca70225640 1 -- 172.21.15.135:0/2298667877 >> v1:172.21.15.194:6789/0 conn(0x7fca68072450 legacy=0x7fca6813c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:41.116 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.116+0000 7fca70225640 1 -- 172.21.15.135:0/2298667877 shutdown_connections 2024-03-29T11:29:41.116 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.116+0000 7fca70225640 1 -- 172.21.15.135:0/2298667877 >> 172.21.15.135:0/2298667877 conn(0x7fca6806b660 msgr2=0x7fca6806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:41.116 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.116+0000 7fca70225640 1 -- 172.21.15.135:0/2298667877 shutdown_connections 2024-03-29T11:29:41.116 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.116+0000 7fca70225640 1 -- 172.21.15.135:0/2298667877 wait complete. 2024-03-29T11:29:41.116 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.117+0000 7fca70225640 10 monclient: build_initial_monmap 2024-03-29T11:29:41.116 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.117+0000 7fca70225640 10 monclient: monmap: 2024-03-29T11:29:41.116 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:29:41.116 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:41.116 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:29:41.116 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:29:41.116 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:41.116 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:41.117 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:29:41.117 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:29:41.117 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:29:41.117 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:41.117 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.118+0000 7fca70225640 1 librados: starting msgr at 2024-03-29T11:29:41.117 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.118+0000 7fca70225640 1 librados: starting objecter 2024-03-29T11:29:41.117 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.118+0000 7fca70225640 1 Processor -- start 2024-03-29T11:29:41.117 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.118+0000 7fca70225640 1 -- start start 2024-03-29T11:29:41.117 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.118+0000 7fca70225640 1 librados: setting wanted keys 2024-03-29T11:29:41.117 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.118+0000 7fca70225640 1 librados: calling monclient init 2024-03-29T11:29:41.117 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.118+0000 7fca70225640 10 monclient: init 2024-03-29T11:29:41.117 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.119+0000 7fca70225640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:41.117 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.119+0000 7fca70225640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:41.118 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.119+0000 7fca70225640 10 monclient(hunting): picked mon.noname-a con 0x7fca68072450 addr v1:172.21.15.135:6789/0 2024-03-29T11:29:41.118 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.119+0000 7fca70225640 10 monclient(hunting): picked mon.noname-b con 0x7fca680b0860 addr v1:172.21.15.194:6789/0 2024-03-29T11:29:41.118 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.119+0000 7fca70225640 10 monclient(hunting): picked mon.noname-c con 0x7fca6813ee40 addr v1:172.21.15.135:6790/0 2024-03-29T11:29:41.118 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.119+0000 7fca70225640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca6806dc20 con 0x7fca68072450 2024-03-29T11:29:41.118 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.119+0000 7fca6e7c2640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7fca6813ee40 0x7fca68070390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:46520/0 (socket says 172.21.15.135:46520) 2024-03-29T11:29:41.118 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.119+0000 7fca6e7c2640 1 -- 172.21.15.135:0/3190899094 learned_addr learned my addr 172.21.15.135:0/3190899094 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:29:41.118 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.119+0000 7fca5e7fc640 10 client.?.objecter ms_handle_connect 0x7fca6813ee40 2024-03-29T11:29:41.118 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.119+0000 7fca5e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:41.118 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.119+0000 7fca5e7fc640 10 client.?.objecter ms_handle_connect 0x7fca680b0860 2024-03-29T11:29:41.118 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.119+0000 7fca5e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:41.118 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.119+0000 7fca70225640 1 -- 172.21.15.135:0/3190899094 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca680b2bd0 con 0x7fca680b0860 2024-03-29T11:29:41.118 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.119+0000 7fca70225640 1 -- 172.21.15.135:0/3190899094 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca68135b80 con 0x7fca6813ee40 2024-03-29T11:29:41.118 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.119+0000 7fca70225640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:41.118 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.119+0000 7fca70225640 10 monclient(hunting): authenticate will time out at 1088.488970s 2024-03-29T11:29:41.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.120+0000 7fca5e7fc640 1 -- 172.21.15.135:0/3190899094 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2969346753 0 0) 0x7fca680b2bd0 con 0x7fca680b0860 2024-03-29T11:29:41.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.120+0000 7fca5e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:41.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.120+0000 7fca5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:41.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.120+0000 7fca5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:41.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.120+0000 7fca5e7fc640 10 monclient(hunting): my global_id is 4417 2024-03-29T11:29:41.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.120+0000 7fca5e7fc640 1 -- 172.21.15.135:0/3190899094 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca4c004eb0 con 0x7fca680b0860 2024-03-29T11:29:41.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.120+0000 7fca5e7fc640 10 client.?.objecter ms_handle_connect 0x7fca68072450 2024-03-29T11:29:41.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.120+0000 7fca5e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:41.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.120+0000 7fca5e7fc640 1 -- 172.21.15.135:0/3190899094 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 497350677 0 0) 0x7fca68135b80 con 0x7fca6813ee40 2024-03-29T11:29:41.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.120+0000 7fca5e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:41.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.120+0000 7fca5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:41.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.120+0000 7fca5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:41.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.120+0000 7fca5e7fc640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:29:41.120 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.120+0000 7fca5e7fc640 1 -- 172.21.15.135:0/3190899094 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca680b2bd0 con 0x7fca6813ee40 2024-03-29T11:29:41.120 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.120+0000 7fca5e7fc640 1 -- 172.21.15.135:0/3190899094 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686648625 0 0) 0x7fca6806dc20 con 0x7fca68072450 2024-03-29T11:29:41.120 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.120+0000 7fca5e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:41.120 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.120+0000 7fca5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:41.120 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.120+0000 7fca5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:41.120 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.120+0000 7fca5e7fc640 10 monclient(hunting): my global_id is 4416 2024-03-29T11:29:41.120 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.120+0000 7fca5e7fc640 1 -- 172.21.15.135:0/3190899094 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca68135b80 con 0x7fca68072450 2024-03-29T11:29:41.120 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.120+0000 7fca5e7fc640 1 -- 172.21.15.135:0/3190899094 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3617198783 0 0) 0x7fca4c004eb0 con 0x7fca680b0860 2024-03-29T11:29:41.120 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.121+0000 7fca5e7fc640 1 -- 172.21.15.135:0/3190899094 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca6806dc20 con 0x7fca680b0860 2024-03-29T11:29:41.120 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.121+0000 7fca5e7fc640 1 -- 172.21.15.135:0/3190899094 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fca60001e80 con 0x7fca680b0860 2024-03-29T11:29:41.120 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.121+0000 7fca5e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:41.120 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.121+0000 7fca5e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:41.120 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.121+0000 7fca5e7fc640 10 monclient(hunting): dump: 2024-03-29T11:29:41.120 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:41.121 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:41.121 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:41.121 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:41.121 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:41.121 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:41.121 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:41.121 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:41.121 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:41.121 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:41.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.121+0000 7fca5e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:41.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.121+0000 7fca5e7fc640 1 -- 172.21.15.135:0/3190899094 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 312464717 0 0) 0x7fca68135b80 con 0x7fca68072450 2024-03-29T11:29:41.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.121+0000 7fca5e7fc640 1 -- 172.21.15.135:0/3190899094 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca4c004eb0 con 0x7fca68072450 2024-03-29T11:29:41.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.121+0000 7fca5e7fc640 1 -- 172.21.15.135:0/3190899094 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fca64001830 con 0x7fca68072450 2024-03-29T11:29:41.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.121+0000 7fca5e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:41.122 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.121+0000 7fca5e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:41.122 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.121+0000 7fca5e7fc640 10 monclient(hunting): dump: 2024-03-29T11:29:41.122 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:41.122 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:41.122 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:41.122 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:41.122 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:41.122 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:41.122 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:41.122 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:41.122 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:41.122 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:41.122 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.121+0000 7fca5e7fc640 1 -- 172.21.15.135:0/3190899094 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4164333716 0 0) 0x7fca680b2bd0 con 0x7fca6813ee40 2024-03-29T11:29:41.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.121+0000 7fca5e7fc640 1 -- 172.21.15.135:0/3190899094 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca68135b80 con 0x7fca6813ee40 2024-03-29T11:29:41.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.121+0000 7fca5e7fc640 1 -- 172.21.15.135:0/3190899094 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2119994083 0 0) 0x7fca6806dc20 con 0x7fca680b0860 2024-03-29T11:29:41.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.121+0000 7fca5e7fc640 1 -- 172.21.15.135:0/3190899094 >> v1:172.21.15.135:6790/0 conn(0x7fca6813ee40 legacy=0x7fca68070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:41.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.121+0000 7fca5e7fc640 1 -- 172.21.15.135:0/3190899094 >> v1:172.21.15.135:6789/0 conn(0x7fca68072450 legacy=0x7fca680b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:41.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.121+0000 7fca5e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:29:41.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.121+0000 7fca5e7fc640 1 monclient: found mon.b 2024-03-29T11:29:41.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.121+0000 7fca5e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:41.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.121+0000 7fca5e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:29:41.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.121+0000 7fca5e7fc640 1 -- 172.21.15.135:0/3190899094 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca68135d10 con 0x7fca680b0860 2024-03-29T11:29:41.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.121+0000 7fca5e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:29:41.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.121+0000 7fca5e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:29:41.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.121+0000 7fca5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:41.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.121+0000 7fca70225640 5 monclient: authenticate success, global_id 4417 2024-03-29T11:29:41.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.122+0000 7fca70225640 10 monclient: _renew_subs 2024-03-29T11:29:41.124 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.122+0000 7fca70225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:29:41.124 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.122+0000 7fca70225640 1 -- 172.21.15.135:0/3190899094 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca680712f0 con 0x7fca680b0860 2024-03-29T11:29:41.124 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.122+0000 7fca70225640 10 client.4417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:41.124 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.122+0000 7fca70225640 10 monclient: _renew_subs 2024-03-29T11:29:41.124 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.122+0000 7fca70225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:29:41.124 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.122+0000 7fca70225640 1 -- 172.21.15.135:0/3190899094 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca680b2020 con 0x7fca680b0860 2024-03-29T11:29:41.124 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.122+0000 7fca70225640 1 librados: init done 2024-03-29T11:29:41.124 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.123+0000 7fca5e7fc640 1 -- 172.21.15.135:0/3190899094 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca60004e10 con 0x7fca680b0860 2024-03-29T11:29:41.124 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.123+0000 7fca5e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:41.124 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.123+0000 7fca5e7fc640 1 -- 172.21.15.135:0/3190899094 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fca60004f80 con 0x7fca680b0860 2024-03-29T11:29:41.124 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.123+0000 7fca5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:41.124 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.123+0000 7fca5e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:41.124 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.123+0000 7fca5e7fc640 10 monclient: dump: 2024-03-29T11:29:41.124 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:41.125 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:41.125 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:41.125 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:41.125 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:41.125 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:41.125 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:41.125 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:41.125 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:41.125 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:41.125 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.123+0000 7fca5ffff640 4 set_mon_vals no callback set 2024-03-29T11:29:41.126 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.124+0000 7fca70225640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:41.126 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.125+0000 7fca5e7fc640 1 -- 172.21.15.135:0/3190899094 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fca6001af50 con 0x7fca680b0860 2024-03-29T11:29:41.127 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.126+0000 7fca70225640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:41.127 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.126+0000 7fca70225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:29:41.127 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.126+0000 7fca70225640 1 -- 172.21.15.135:0/3190899094 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca50005080 con 0x7fca680b0860 2024-03-29T11:29:41.127 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.126+0000 7fca5e7fc640 1 -- 172.21.15.135:0/3190899094 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7fca60084bb0 con 0x7fca680b0860 2024-03-29T11:29:41.127 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.126+0000 7fca5e7fc640 10 client.4417.objecter ms_dispatch 0x7fca68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:41.127 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.126+0000 7fca5e7fc640 3 client.4417.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:41.127 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.126+0000 7fca5e7fc640 3 client.4417.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:41.127 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.127+0000 7fca6dfc1640 10 monclient: get_auth_request con 0x7fca4c06e090 auth_method 0 2024-03-29T11:29:41.127 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.127+0000 7fca5e7fc640 20 client.4417.objecter dump_active .. 0 homeless 2024-03-29T11:29:41.127 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.127+0000 7fca5e7fc640 10 client.4417.objecter ms_handle_connect 0x7fca4c06e090 2024-03-29T11:29:41.127 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.133+0000 7fca5e7fc640 1 -- 172.21.15.135:0/3190899094 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca60050490 con 0x7fca680b0860 2024-03-29T11:29:41.127 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.133+0000 7fca5e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:41.127 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.133+0000 7fca5e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:41.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.510+0000 7fca70225640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:41.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.510+0000 7fca70225640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:41.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.510+0000 7fca70225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:29:41.503 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.510+0000 7fca70225640 1 -- 172.21.15.135:0/3190899094 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca50005330 con 0x7fca680b0860 2024-03-29T11:29:41.504 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.511+0000 7fca5e7fc640 1 -- 172.21.15.135:0/3190899094 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7fca600523c0 con 0x7fca680b0860 2024-03-29T11:29:41.504 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.511+0000 7fca5e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:41.504 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.511+0000 7fca5e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:41.504 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:29:41.504 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:41.510 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.517+0000 7fca70225640 10 librados: watch_flush enter 2024-03-29T11:29:41.510 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.518+0000 7fca70225640 10 librados: watch_flush exit 2024-03-29T11:29:41.510 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.518+0000 7fca70225640 20 client.4417.objecter shutdown clearing up homeless session... 2024-03-29T11:29:41.510 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.518+0000 7fca70225640 10 client.4417.objecter successfully canceled tick 2024-03-29T11:29:41.510 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.518+0000 7fca70225640 1 -- 172.21.15.135:0/3190899094 >> v1:172.21.15.135:6816/3469875285 conn(0x7fca4c06e090 legacy=0x7fca4c070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:41.510 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.518+0000 7fca70225640 10 monclient: shutdown 2024-03-29T11:29:41.510 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.518+0000 7fca70225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:41.510 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.518+0000 7fca70225640 1 -- 172.21.15.135:0/3190899094 >> v1:172.21.15.194:6789/0 conn(0x7fca680b0860 legacy=0x7fca681db830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:41.511 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.518+0000 7fca70225640 1 -- 172.21.15.135:0/3190899094 shutdown_connections 2024-03-29T11:29:41.511 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.518+0000 7fca70225640 1 -- 172.21.15.135:0/3190899094 >> 172.21.15.135:0/3190899094 conn(0x7fca6806b660 msgr2=0x7fca6813bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:41.511 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.518+0000 7fca70225640 1 -- 172.21.15.135:0/3190899094 shutdown_connections 2024-03-29T11:29:41.511 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.518+0000 7fca70225640 1 -- 172.21.15.135:0/3190899094 wait complete. 2024-03-29T11:29:41.511 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:41.519+0000 7fca70225640 1 librados: shutdown 2024-03-29T11:29:41.529 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:48.335 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:48.510 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.517+0000 7f7b859cd640 10 monclient: get_monmap_and_config 2024-03-29T11:29:48.510 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.517+0000 7f7b859cd640 10 monclient: build_initial_monmap 2024-03-29T11:29:48.510 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.517+0000 7f7b859cd640 10 monclient: monmap: 2024-03-29T11:29:48.510 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:29:48.510 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:48.510 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:29:48.519432+0000 2024-03-29T11:29:48.511 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:29:48.519432+0000 2024-03-29T11:29:48.511 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:48.511 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:48.511 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:29:48.511 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:29:48.511 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:29:48.511 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:48.511 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.519+0000 7f7b859cd640 1 Processor -- start 2024-03-29T11:29:48.511 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.519+0000 7f7b859cd640 1 -- start start 2024-03-29T11:29:48.512 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.519+0000 7f7b859cd640 10 monclient: init 2024-03-29T11:29:48.513 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.521+0000 7f7b859cd640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:48.513 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.521+0000 7f7b859cd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:48.514 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.521+0000 7f7b859cd640 10 monclient(hunting): picked mon.noname-c con 0x7f7b801457f0 addr v1:172.21.15.194:6789/0 2024-03-29T11:29:48.514 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.521+0000 7f7b859cd640 10 monclient(hunting): picked mon.noname-b con 0x7f7b8013f650 addr v1:172.21.15.135:6790/0 2024-03-29T11:29:48.514 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.521+0000 7f7b859cd640 10 monclient(hunting): picked mon.noname-a con 0x7f7b800a9d70 addr v1:172.21.15.135:6789/0 2024-03-29T11:29:48.514 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.521+0000 7f7b859cd640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b800aaef0 con 0x7f7b800a9d70 2024-03-29T11:29:48.514 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.521+0000 7f7b859cd640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b80145bd0 con 0x7f7b801457f0 2024-03-29T11:29:48.514 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.521+0000 7f7b7effd640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f7b801457f0 0x7f7b8013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:49472/0 (socket says 172.21.15.135:49472) 2024-03-29T11:29:48.514 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.521+0000 7f7b7effd640 1 -- 172.21.15.135:0/1814627592 learned_addr learned my addr 172.21.15.135:0/1814627592 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:29:48.514 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.521+0000 7f7b859cd640 1 -- 172.21.15.135:0/1814627592 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b80145eb0 con 0x7f7b8013f650 2024-03-29T11:29:48.514 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.521+0000 7f7b859cd640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:48.514 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.521+0000 7f7b859cd640 10 monclient(hunting): authenticate will time out at 1095.890543s 2024-03-29T11:29:48.515 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 1 -- 172.21.15.135:0/1814627592 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586321378 0 0) 0x7f7b80145bd0 con 0x7f7b801457f0 2024-03-29T11:29:48.515 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:48.515 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:48.516 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:48.516 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:29:48.516 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 1 -- 172.21.15.135:0/1814627592 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b68003520 con 0x7f7b801457f0 2024-03-29T11:29:48.516 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 1 -- 172.21.15.135:0/1814627592 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1237421708 0 0) 0x7f7b80145eb0 con 0x7f7b8013f650 2024-03-29T11:29:48.516 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:48.516 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:48.516 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:48.516 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:29:48.516 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 1 -- 172.21.15.135:0/1814627592 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b80145bd0 con 0x7f7b8013f650 2024-03-29T11:29:48.516 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 1 -- 172.21.15.135:0/1814627592 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3215786901 0 0) 0x7f7b80145bd0 con 0x7f7b8013f650 2024-03-29T11:29:48.516 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 1 -- 172.21.15.135:0/1814627592 >> v1:172.21.15.194:6789/0 conn(0x7f7b801457f0 legacy=0x7f7b8013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:48.516 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 1 -- 172.21.15.135:0/1814627592 >> v1:172.21.15.135:6789/0 conn(0x7f7b800a9d70 legacy=0x7f7b8014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:48.516 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:29:48.516 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 1 monclient: found mon.noname-b 2024-03-29T11:29:48.517 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:48.517 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:29:48.517 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 1 -- 172.21.15.135:0/1814627592 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b80146340 con 0x7f7b8013f650 2024-03-29T11:29:48.517 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 10 monclient: _finish_auth 0 2024-03-29T11:29:48.517 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:29:48.517 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:48.517 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b859cd640 5 monclient: authenticate success, global_id 4424 2024-03-29T11:29:48.517 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 1 -- 172.21.15.135:0/1814627592 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f7b70001670 con 0x7f7b8013f650 2024-03-29T11:29:48.517 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b859cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:48.517 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:48.517 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:48.517 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b7dffb640 10 monclient: dump: 2024-03-29T11:29:48.517 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:48.517 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:48.517 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:48.518 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:48.518 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:48.518 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:48.518 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:48.518 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:48.518 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:48.518 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:48.518 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.522+0000 7f7b859cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:48.519 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.524+0000 7f7b7dffb640 1 -- 172.21.15.135:0/1814627592 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b70001890 con 0x7f7b8013f650 2024-03-29T11:29:48.519 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.524+0000 7f7b7dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:48.519 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.524+0000 7f7b859cd640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:48.519 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.524+0000 7f7b859cd640 4 set_mon_vals no callback set 2024-03-29T11:29:48.520 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.524+0000 7f7b7dffb640 1 -- 172.21.15.135:0/1814627592 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f7b70001c20 con 0x7f7b8013f650 2024-03-29T11:29:48.520 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.524+0000 7f7b7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:48.520 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.524+0000 7f7b7dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:48.520 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.524+0000 7f7b7dffb640 10 monclient: dump: 2024-03-29T11:29:48.520 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:48.520 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:48.520 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:48.520 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:48.520 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:48.520 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:48.520 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:48.520 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:48.520 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:48.520 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.525+0000 7f7b859cd640 10 monclient: shutdown 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.525+0000 7f7b859cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.525+0000 7f7b859cd640 1 -- 172.21.15.135:0/1814627592 >> v1:172.21.15.135:6790/0 conn(0x7f7b8013f650 legacy=0x7f7b800b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.525+0000 7f7b859cd640 1 -- 172.21.15.135:0/1814627592 shutdown_connections 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.525+0000 7f7b859cd640 1 -- 172.21.15.135:0/1814627592 >> 172.21.15.135:0/1814627592 conn(0x7f7b80066a00 msgr2=0x7f7b80066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.525+0000 7f7b859cd640 1 -- 172.21.15.135:0/1814627592 shutdown_connections 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.525+0000 7f7b859cd640 1 -- 172.21.15.135:0/1814627592 wait complete. 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.526+0000 7f7b859cd640 10 monclient: build_initial_monmap 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.526+0000 7f7b859cd640 10 monclient: monmap: 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:48.521 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:48.522 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:29:48.522 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:29:48.522 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:29:48.522 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:48.522 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.526+0000 7f7b859cd640 1 librados: starting msgr at 2024-03-29T11:29:48.522 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.526+0000 7f7b859cd640 1 librados: starting objecter 2024-03-29T11:29:48.522 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.526+0000 7f7b859cd640 1 Processor -- start 2024-03-29T11:29:48.522 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.527+0000 7f7b859cd640 1 -- start start 2024-03-29T11:29:48.522 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.527+0000 7f7b859cd640 1 librados: setting wanted keys 2024-03-29T11:29:48.522 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.527+0000 7f7b859cd640 1 librados: calling monclient init 2024-03-29T11:29:48.522 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.527+0000 7f7b859cd640 10 monclient: init 2024-03-29T11:29:48.529 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.528+0000 7f7b859cd640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:48.529 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.528+0000 7f7b859cd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:48.529 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.528+0000 7f7b859cd640 10 monclient(hunting): picked mon.noname-a con 0x7f7b800a9d70 addr v1:172.21.15.135:6789/0 2024-03-29T11:29:48.529 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.528+0000 7f7b859cd640 10 monclient(hunting): picked mon.noname-c con 0x7f7b8013f650 addr v1:172.21.15.135:6790/0 2024-03-29T11:29:48.529 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.528+0000 7f7b859cd640 10 monclient(hunting): picked mon.noname-b con 0x7f7b801457f0 addr v1:172.21.15.194:6789/0 2024-03-29T11:29:48.529 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.528+0000 7f7b859cd640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b800719e0 con 0x7f7b800a9d70 2024-03-29T11:29:48.529 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.528+0000 7f7b859cd640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b800ac140 con 0x7f7b801457f0 2024-03-29T11:29:48.529 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.528+0000 7f7b859cd640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b800ac420 con 0x7f7b8013f650 2024-03-29T11:29:48.529 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.528+0000 7f7b859cd640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:48.529 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.528+0000 7f7b859cd640 10 monclient(hunting): authenticate will time out at 1095.897221s 2024-03-29T11:29:48.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.528+0000 7f7b7f7fe640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f7b801457f0 0x7f7b801e6020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:49886/0 (socket says 172.21.15.135:49886) 2024-03-29T11:29:48.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.528+0000 7f7b7f7fe640 1 -- 172.21.15.135:0/2111142794 learned_addr learned my addr 172.21.15.135:0/2111142794 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:29:48.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.528+0000 7f7b67fff640 10 client.?.objecter ms_handle_connect 0x7f7b800a9d70 2024-03-29T11:29:48.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.528+0000 7f7b67fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:48.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.528+0000 7f7b67fff640 10 client.?.objecter ms_handle_connect 0x7f7b801457f0 2024-03-29T11:29:48.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.528+0000 7f7b67fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:48.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.529+0000 7f7b67fff640 10 client.?.objecter ms_handle_connect 0x7f7b8013f650 2024-03-29T11:29:48.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.529+0000 7f7b67fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:48.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.529+0000 7f7b67fff640 1 -- 172.21.15.135:0/2111142794 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652926733 0 0) 0x7f7b800ac140 con 0x7f7b801457f0 2024-03-29T11:29:48.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.529+0000 7f7b67fff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:48.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.529+0000 7f7b67fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:48.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.529+0000 7f7b67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:48.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.529+0000 7f7b67fff640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:29:48.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.529+0000 7f7b67fff640 1 -- 172.21.15.135:0/2111142794 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b6c004eb0 con 0x7f7b801457f0 2024-03-29T11:29:48.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.529+0000 7f7b67fff640 1 -- 172.21.15.135:0/2111142794 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986056658 0 0) 0x7f7b800719e0 con 0x7f7b800a9d70 2024-03-29T11:29:48.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.529+0000 7f7b67fff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:48.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.529+0000 7f7b67fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:48.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.529+0000 7f7b67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:48.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.529+0000 7f7b67fff640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:29:48.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.529+0000 7f7b67fff640 1 -- 172.21.15.135:0/2111142794 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b800ac140 con 0x7f7b800a9d70 2024-03-29T11:29:48.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.529+0000 7f7b67fff640 1 -- 172.21.15.135:0/2111142794 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2698813148 0 0) 0x7f7b800ac420 con 0x7f7b8013f650 2024-03-29T11:29:48.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.529+0000 7f7b67fff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:48.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.529+0000 7f7b67fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:48.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.529+0000 7f7b67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:48.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.529+0000 7f7b67fff640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:29:48.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.529+0000 7f7b67fff640 1 -- 172.21.15.135:0/2111142794 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b800719e0 con 0x7f7b8013f650 2024-03-29T11:29:48.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.530+0000 7f7b67fff640 1 -- 172.21.15.135:0/2111142794 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2566808668 0 0) 0x7f7b800719e0 con 0x7f7b8013f650 2024-03-29T11:29:48.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.530+0000 7f7b67fff640 1 -- 172.21.15.135:0/2111142794 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b800ac420 con 0x7f7b8013f650 2024-03-29T11:29:48.532 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.530+0000 7f7b67fff640 1 -- 172.21.15.135:0/2111142794 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f7b700015b0 con 0x7f7b8013f650 2024-03-29T11:29:48.532 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.530+0000 7f7b67fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:48.532 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.530+0000 7f7b67fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:48.532 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.530+0000 7f7b67fff640 10 monclient(hunting): dump: 2024-03-29T11:29:48.532 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:48.532 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:48.532 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:48.532 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:48.532 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:48.532 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:48.532 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:48.532 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:48.532 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:48.532 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:48.532 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.530+0000 7f7b67fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:48.533 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.530+0000 7f7b67fff640 1 -- 172.21.15.135:0/2111142794 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1931613790 0 0) 0x7f7b6c004eb0 con 0x7f7b801457f0 2024-03-29T11:29:48.533 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.530+0000 7f7b67fff640 1 -- 172.21.15.135:0/2111142794 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b800719e0 con 0x7f7b801457f0 2024-03-29T11:29:48.533 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.530+0000 7f7b67fff640 1 -- 172.21.15.135:0/2111142794 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f7b78001d30 con 0x7f7b801457f0 2024-03-29T11:29:48.533 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.530+0000 7f7b67fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:48.533 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.530+0000 7f7b67fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:48.533 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.530+0000 7f7b67fff640 10 monclient(hunting): dump: 2024-03-29T11:29:48.533 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:48.533 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:48.533 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:48.533 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:48.533 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:48.533 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:48.533 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:48.533 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:48.534 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:48.534 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:48.534 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.530+0000 7f7b67fff640 1 -- 172.21.15.135:0/2111142794 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2904128737 0 0) 0x7f7b800ac140 con 0x7f7b800a9d70 2024-03-29T11:29:48.534 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.530+0000 7f7b67fff640 1 -- 172.21.15.135:0/2111142794 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b6c004eb0 con 0x7f7b800a9d70 2024-03-29T11:29:48.534 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.530+0000 7f7b67fff640 1 -- 172.21.15.135:0/2111142794 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f7b740016f0 con 0x7f7b800a9d70 2024-03-29T11:29:48.534 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.530+0000 7f7b67fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:48.534 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.530+0000 7f7b67fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:48.534 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.530+0000 7f7b67fff640 10 monclient(hunting): dump: 2024-03-29T11:29:48.534 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:48.534 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:48.534 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:48.534 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:48.534 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:48.534 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:48.535 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:48.535 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:48.535 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:48.535 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:48.535 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.531+0000 7f7b67fff640 1 -- 172.21.15.135:0/2111142794 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4155466172 0 0) 0x7f7b800ac420 con 0x7f7b8013f650 2024-03-29T11:29:48.537 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.531+0000 7f7b67fff640 1 -- 172.21.15.135:0/2111142794 >> v1:172.21.15.194:6789/0 conn(0x7f7b801457f0 legacy=0x7f7b801e6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:48.537 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.531+0000 7f7b67fff640 1 -- 172.21.15.135:0/2111142794 >> v1:172.21.15.135:6789/0 conn(0x7f7b800a9d70 legacy=0x7f7b80070cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:48.537 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.531+0000 7f7b67fff640 10 monclient: _finish_hunting 0 2024-03-29T11:29:48.538 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.531+0000 7f7b67fff640 1 monclient: found mon.c 2024-03-29T11:29:48.538 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.531+0000 7f7b67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:48.538 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.531+0000 7f7b67fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:29:48.538 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.531+0000 7f7b67fff640 1 -- 172.21.15.135:0/2111142794 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b80071500 con 0x7f7b8013f650 2024-03-29T11:29:48.538 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.531+0000 7f7b67fff640 10 monclient: _finish_auth 0 2024-03-29T11:29:48.538 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.531+0000 7f7b67fff640 10 monclient: _check_auth_tickets 2024-03-29T11:29:48.538 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.531+0000 7f7b67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:48.538 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.531+0000 7f7b859cd640 5 monclient: authenticate success, global_id 4427 2024-03-29T11:29:48.538 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.531+0000 7f7b859cd640 10 monclient: _renew_subs 2024-03-29T11:29:48.538 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.531+0000 7f7b859cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:29:48.538 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.531+0000 7f7b859cd640 1 -- 172.21.15.135:0/2111142794 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b80071e70 con 0x7f7b8013f650 2024-03-29T11:29:48.538 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.531+0000 7f7b859cd640 10 client.4427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:48.538 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.531+0000 7f7b859cd640 10 monclient: _renew_subs 2024-03-29T11:29:48.538 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.531+0000 7f7b859cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:29:48.539 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.531+0000 7f7b859cd640 1 -- 172.21.15.135:0/2111142794 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b8006e9e0 con 0x7f7b8013f650 2024-03-29T11:29:48.539 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.531+0000 7f7b859cd640 1 librados: init done 2024-03-29T11:29:48.539 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.531+0000 7f7b67fff640 1 -- 172.21.15.135:0/2111142794 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b70001710 con 0x7f7b8013f650 2024-03-29T11:29:48.539 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.531+0000 7f7b67fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:48.539 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.531+0000 7f7b7d7fa640 4 set_mon_vals no callback set 2024-03-29T11:29:48.539 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.532+0000 7f7b67fff640 1 -- 172.21.15.135:0/2111142794 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f7b70003a40 con 0x7f7b8013f650 2024-03-29T11:29:48.539 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.532+0000 7f7b67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:48.539 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.532+0000 7f7b67fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:48.539 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.532+0000 7f7b67fff640 10 monclient: dump: 2024-03-29T11:29:48.539 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:48.539 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:48.539 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:48.539 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:48.539 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:48.539 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:48.540 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:48.540 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:48.540 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:48.540 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:48.540 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.533+0000 7f7b859cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:48.540 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.533+0000 7f7b859cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:48.540 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.533+0000 7f7b859cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:29:48.540 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.534+0000 7f7b859cd640 1 -- 172.21.15.135:0/2111142794 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b60005080 con 0x7f7b8013f650 2024-03-29T11:29:48.540 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.534+0000 7f7b67fff640 1 -- 172.21.15.135:0/2111142794 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f7b70019a10 con 0x7f7b8013f650 2024-03-29T11:29:48.540 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.535+0000 7f7b67fff640 1 -- 172.21.15.135:0/2111142794 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f7b70083770 con 0x7f7b8013f650 2024-03-29T11:29:48.540 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.535+0000 7f7b67fff640 10 client.4427.objecter ms_dispatch 0x7f7b80000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:48.540 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.535+0000 7f7b67fff640 3 client.4427.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:48.540 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.535+0000 7f7b67fff640 3 client.4427.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:48.540 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.535+0000 7f7b7effd640 10 monclient: get_auth_request con 0x7f7b6c06e080 auth_method 0 2024-03-29T11:29:48.540 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.535+0000 7f7b67fff640 20 client.4427.objecter dump_active .. 0 homeless 2024-03-29T11:29:48.540 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.536+0000 7f7b67fff640 10 client.4427.objecter ms_handle_connect 0x7f7b6c06e080 2024-03-29T11:29:48.541 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.545+0000 7f7b67fff640 1 -- 172.21.15.135:0/2111142794 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b7004f050 con 0x7f7b8013f650 2024-03-29T11:29:48.541 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.545+0000 7f7b67fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:48.541 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.545+0000 7f7b67fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:48.972 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.980+0000 7f7b859cd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:48.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.980+0000 7f7b859cd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:48.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.980+0000 7f7b859cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:29:48.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.980+0000 7f7b859cd640 1 -- 172.21.15.135:0/2111142794 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b60005330 con 0x7f7b8013f650 2024-03-29T11:29:48.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.981+0000 7f7b67fff640 1 -- 172.21.15.135:0/2111142794 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f7b70056ba0 con 0x7f7b8013f650 2024-03-29T11:29:48.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.981+0000 7f7b67fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:48.974 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.981+0000 7f7b67fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:48.974 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:29:48.974 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:48.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.985+0000 7f7b859cd640 10 librados: watch_flush enter 2024-03-29T11:29:48.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.985+0000 7f7b859cd640 10 librados: watch_flush exit 2024-03-29T11:29:48.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.985+0000 7f7b859cd640 20 client.4427.objecter shutdown clearing up homeless session... 2024-03-29T11:29:48.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.985+0000 7f7b859cd640 10 client.4427.objecter successfully canceled tick 2024-03-29T11:29:48.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.986+0000 7f7b859cd640 1 -- 172.21.15.135:0/2111142794 >> v1:172.21.15.135:6816/3469875285 conn(0x7f7b6c06e080 legacy=0x7f7b6c070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:48.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.986+0000 7f7b859cd640 10 monclient: shutdown 2024-03-29T11:29:48.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.986+0000 7f7b859cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:48.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.986+0000 7f7b859cd640 1 -- 172.21.15.135:0/2111142794 >> v1:172.21.15.135:6790/0 conn(0x7f7b8013f650 legacy=0x7f7b8006e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:48.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.986+0000 7f7b859cd640 1 -- 172.21.15.135:0/2111142794 shutdown_connections 2024-03-29T11:29:48.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.986+0000 7f7b859cd640 1 -- 172.21.15.135:0/2111142794 >> 172.21.15.135:0/2111142794 conn(0x7f7b80066a00 msgr2=0x7f7b800b5790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:48.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.986+0000 7f7b859cd640 1 -- 172.21.15.135:0/2111142794 shutdown_connections 2024-03-29T11:29:48.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.986+0000 7f7b859cd640 1 -- 172.21.15.135:0/2111142794 wait complete. 2024-03-29T11:29:48.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:48.986+0000 7f7b859cd640 1 librados: shutdown 2024-03-29T11:29:48.995 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:55.802 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:55.953 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.961+0000 7f224679d640 10 monclient: get_monmap_and_config 2024-03-29T11:29:55.953 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.961+0000 7f224679d640 10 monclient: build_initial_monmap 2024-03-29T11:29:55.954 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.961+0000 7f224679d640 10 monclient: monmap: 2024-03-29T11:29:55.954 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:29:55.954 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:55.954 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:29:55.962669+0000 2024-03-29T11:29:55.954 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:29:55.962669+0000 2024-03-29T11:29:55.954 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:55.954 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:55.954 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:29:55.954 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:29:55.954 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:29:55.954 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:55.955 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.962+0000 7f224679d640 1 Processor -- start 2024-03-29T11:29:55.955 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.962+0000 7f224679d640 1 -- start start 2024-03-29T11:29:55.955 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.962+0000 7f224679d640 10 monclient: init 2024-03-29T11:29:55.957 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.964+0000 7f224679d640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:55.957 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.964+0000 7f224679d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:55.957 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.964+0000 7f224679d640 10 monclient(hunting): picked mon.noname-c con 0x7f22401457f0 addr v1:172.21.15.194:6789/0 2024-03-29T11:29:55.957 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.964+0000 7f224679d640 10 monclient(hunting): picked mon.noname-a con 0x7f224013f650 addr v1:172.21.15.135:6789/0 2024-03-29T11:29:55.957 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.964+0000 7f224679d640 10 monclient(hunting): picked mon.noname-b con 0x7f22400a9d70 addr v1:172.21.15.135:6790/0 2024-03-29T11:29:55.957 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.964+0000 7f224679d640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22400aaef0 con 0x7f224013f650 2024-03-29T11:29:55.957 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.964+0000 7f224679d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2240145bd0 con 0x7f22401457f0 2024-03-29T11:29:55.957 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.964+0000 7f224679d640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2240145eb0 con 0x7f22400a9d70 2024-03-29T11:29:55.957 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.964+0000 7f224679d640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:55.957 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.964+0000 7f224679d640 10 monclient(hunting): authenticate will time out at 1103.333824s 2024-03-29T11:29:55.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.967+0000 7f223ffff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f22401457f0 0x7f224013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:50628/0 (socket says 172.21.15.135:50628) 2024-03-29T11:29:55.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.967+0000 7f223ffff640 1 -- 172.21.15.135:0/4050854291 learned_addr learned my addr 172.21.15.135:0/4050854291 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:29:55.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 1 -- 172.21.15.135:0/4050854291 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1362068791 0 0) 0x7f22400aaef0 con 0x7f224013f650 2024-03-29T11:29:55.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:55.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:55.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:55.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:29:55.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 1 -- 172.21.15.135:0/4050854291 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2228003520 con 0x7f224013f650 2024-03-29T11:29:55.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 1 -- 172.21.15.135:0/4050854291 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1933171910 0 0) 0x7f2240145eb0 con 0x7f22400a9d70 2024-03-29T11:29:55.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:55.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:55.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:55.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:29:55.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 1 -- 172.21.15.135:0/4050854291 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22400aaef0 con 0x7f22400a9d70 2024-03-29T11:29:55.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 1 -- 172.21.15.135:0/4050854291 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3734320753 0 0) 0x7f22400aaef0 con 0x7f22400a9d70 2024-03-29T11:29:55.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 1 -- 172.21.15.135:0/4050854291 >> v1:172.21.15.194:6789/0 conn(0x7f22401457f0 legacy=0x7f224013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:55.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 1 -- 172.21.15.135:0/4050854291 >> v1:172.21.15.135:6789/0 conn(0x7f224013f650 legacy=0x7f22400b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:55.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:55.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 1 monclient: found mon.noname-b 2024-03-29T11:29:55.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:55.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:29:55.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 1 -- 172.21.15.135:0/4050854291 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2240146340 con 0x7f22400a9d70 2024-03-29T11:29:55.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 10 monclient: _finish_auth 0 2024-03-29T11:29:55.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:55.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:55.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 1 -- 172.21.15.135:0/4050854291 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f22380019f0 con 0x7f22400a9d70 2024-03-29T11:29:55.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f224679d640 5 monclient: authenticate success, global_id 4430 2024-03-29T11:29:55.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f224679d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:55.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:55.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:55.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 10 monclient: dump: 2024-03-29T11:29:55.967 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:55.967 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:55.967 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:55.967 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:55.967 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:55.967 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:55.967 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:55.968 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:55.968 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:55.968 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:55.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f224679d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:55.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 1 -- 172.21.15.135:0/4050854291 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2238001c10 con 0x7f22400a9d70 2024-03-29T11:29:55.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f223effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:55.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f224679d640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:55.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.968+0000 7f224679d640 4 set_mon_vals no callback set 2024-03-29T11:29:55.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.969+0000 7f223effd640 1 -- 172.21.15.135:0/4050854291 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f2238001fa0 con 0x7f22400a9d70 2024-03-29T11:29:55.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.969+0000 7f223effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:55.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.969+0000 7f223effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:55.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.969+0000 7f223effd640 10 monclient: dump: 2024-03-29T11:29:55.968 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:55.969 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:55.969 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:55.969 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:55.969 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:55.969 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:55.969 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:55.969 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:55.969 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:55.969 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:55.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.969+0000 7f224679d640 10 monclient: shutdown 2024-03-29T11:29:55.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.969+0000 7f224679d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:55.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.969+0000 7f224679d640 1 -- 172.21.15.135:0/4050854291 >> v1:172.21.15.135:6790/0 conn(0x7f22400a9d70 legacy=0x7f224014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:55.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.969+0000 7f224679d640 1 -- 172.21.15.135:0/4050854291 shutdown_connections 2024-03-29T11:29:55.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.969+0000 7f224679d640 1 -- 172.21.15.135:0/4050854291 >> 172.21.15.135:0/4050854291 conn(0x7f2240066a00 msgr2=0x7f2240066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:55.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.970+0000 7f224679d640 1 -- 172.21.15.135:0/4050854291 shutdown_connections 2024-03-29T11:29:55.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.970+0000 7f224679d640 1 -- 172.21.15.135:0/4050854291 wait complete. 2024-03-29T11:29:55.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.971+0000 7f224679d640 10 monclient: build_initial_monmap 2024-03-29T11:29:55.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.971+0000 7f224679d640 10 monclient: monmap: 2024-03-29T11:29:55.970 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:29:55.970 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:55.970 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:29:55.970 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:29:55.970 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:55.970 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:55.970 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:29:55.970 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:29:55.970 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:29:55.970 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:55.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.972+0000 7f224679d640 1 librados: starting msgr at 2024-03-29T11:29:55.971 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.972+0000 7f224679d640 1 librados: starting objecter 2024-03-29T11:29:55.971 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.972+0000 7f224679d640 1 Processor -- start 2024-03-29T11:29:55.971 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.972+0000 7f224679d640 1 -- start start 2024-03-29T11:29:55.971 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.972+0000 7f224679d640 1 librados: setting wanted keys 2024-03-29T11:29:55.971 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.972+0000 7f224679d640 1 librados: calling monclient init 2024-03-29T11:29:55.971 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.972+0000 7f224679d640 10 monclient: init 2024-03-29T11:29:55.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.973+0000 7f224679d640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:55.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.973+0000 7f224679d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:55.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.973+0000 7f224679d640 10 monclient(hunting): picked mon.noname-c con 0x7f22400a9d70 addr v1:172.21.15.135:6790/0 2024-03-29T11:29:55.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.973+0000 7f224679d640 10 monclient(hunting): picked mon.noname-b con 0x7f224013f650 addr v1:172.21.15.194:6789/0 2024-03-29T11:29:55.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.973+0000 7f224679d640 10 monclient(hunting): picked mon.noname-a con 0x7f22401457f0 addr v1:172.21.15.135:6789/0 2024-03-29T11:29:55.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.973+0000 7f224679d640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f224013e4c0 con 0x7f22401457f0 2024-03-29T11:29:55.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.973+0000 7f224679d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f224013e7a0 con 0x7f224013f650 2024-03-29T11:29:55.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.973+0000 7f224679d640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f224013c9d0 con 0x7f22400a9d70 2024-03-29T11:29:55.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.973+0000 7f224679d640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:55.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.973+0000 7f224679d640 10 monclient(hunting): authenticate will time out at 1103.342416s 2024-03-29T11:29:55.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.974+0000 7f223f7fe640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f224013f650 0x7f2240143620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:45178/0 (socket says 172.21.15.135:45178) 2024-03-29T11:29:55.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.974+0000 7f223f7fe640 1 -- 172.21.15.135:0/2968054451 learned_addr learned my addr 172.21.15.135:0/2968054451 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:29:55.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.974+0000 7f223cff9640 10 client.?.objecter ms_handle_connect 0x7f224013f650 2024-03-29T11:29:55.974 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.974+0000 7f223cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:55.974 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.976+0000 7f223cff9640 1 -- 172.21.15.135:0/2968054451 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2780714581 0 0) 0x7f224013e7a0 con 0x7f224013f650 2024-03-29T11:29:55.974 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.976+0000 7f223cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:55.974 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.976+0000 7f223cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:55.974 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.976+0000 7f223cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:55.974 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.976+0000 7f223cff9640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:29:55.974 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.976+0000 7f223cff9640 1 -- 172.21.15.135:0/2968054451 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f222c004eb0 con 0x7f224013f650 2024-03-29T11:29:55.974 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.976+0000 7f223cff9640 10 client.?.objecter ms_handle_connect 0x7f22400a9d70 2024-03-29T11:29:55.974 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.976+0000 7f223cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:55.974 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.976+0000 7f223cff9640 1 -- 172.21.15.135:0/2968054451 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2324663358 0 0) 0x7f222c004eb0 con 0x7f224013f650 2024-03-29T11:29:55.974 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.976+0000 7f223cff9640 1 -- 172.21.15.135:0/2968054451 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f224013e7a0 con 0x7f224013f650 2024-03-29T11:29:55.974 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.976+0000 7f223cff9640 1 -- 172.21.15.135:0/2968054451 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f2234001a30 con 0x7f224013f650 2024-03-29T11:29:55.974 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.976+0000 7f223cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:55.974 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.976+0000 7f223cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:55.975 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.976+0000 7f223cff9640 10 monclient(hunting): dump: 2024-03-29T11:29:55.975 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:55.975 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:55.975 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:55.975 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:55.975 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:55.975 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:55.975 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:55.975 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:55.975 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:55.975 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:55.975 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.976+0000 7f223cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:55.975 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.978+0000 7f223cff9640 1 -- 172.21.15.135:0/2968054451 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 478476995 0 0) 0x7f224013e7a0 con 0x7f224013f650 2024-03-29T11:29:55.975 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.978+0000 7f223cff9640 1 -- 172.21.15.135:0/2968054451 >> v1:172.21.15.135:6790/0 conn(0x7f22400a9d70 legacy=0x7f22401362f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:55.976 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.978+0000 7f223cff9640 1 -- 172.21.15.135:0/2968054451 >> v1:172.21.15.135:6789/0 conn(0x7f22401457f0 legacy=0x7f22401ee860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:55.976 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.978+0000 7f223cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:29:55.976 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.978+0000 7f223cff9640 1 monclient: found mon.b 2024-03-29T11:29:55.976 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.978+0000 7f223cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:55.976 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.978+0000 7f223cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:29:55.976 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.978+0000 7f223cff9640 1 -- 172.21.15.135:0/2968054451 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224013ccb0 con 0x7f224013f650 2024-03-29T11:29:55.976 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.978+0000 7f223cff9640 10 monclient: _finish_auth 0 2024-03-29T11:29:55.976 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.978+0000 7f223cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:29:55.976 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.978+0000 7f223cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:55.976 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.978+0000 7f223cff9640 1 -- 172.21.15.135:0/2968054451 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2234001cf0 con 0x7f224013f650 2024-03-29T11:29:55.976 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.978+0000 7f223cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:55.976 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.978+0000 7f223dffb640 4 set_mon_vals no callback set 2024-03-29T11:29:55.976 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.979+0000 7f223cff9640 1 -- 172.21.15.135:0/2968054451 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f2234003a60 con 0x7f224013f650 2024-03-29T11:29:55.976 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.979+0000 7f223cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:55.976 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.979+0000 7f223cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:55.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.979+0000 7f223cff9640 10 monclient: dump: 2024-03-29T11:29:55.977 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:29:55.977 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:29:55.977 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:55.977 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:29:55.977 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:55.977 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:29:55.977 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:29:55.977 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:29:55.977 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:29:55.978 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:29:55.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.979+0000 7f224679d640 5 monclient: authenticate success, global_id 4426 2024-03-29T11:29:55.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.979+0000 7f224679d640 10 monclient: _renew_subs 2024-03-29T11:29:55.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.979+0000 7f224679d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:29:55.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.979+0000 7f224679d640 1 -- 172.21.15.135:0/2968054451 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2240143d30 con 0x7f224013f650 2024-03-29T11:29:55.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.980+0000 7f224679d640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:55.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.980+0000 7f224679d640 10 monclient: _renew_subs 2024-03-29T11:29:55.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.980+0000 7f224679d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:29:55.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.980+0000 7f224679d640 1 -- 172.21.15.135:0/2968054451 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2240133880 con 0x7f224013f650 2024-03-29T11:29:55.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.980+0000 7f224679d640 1 librados: init done 2024-03-29T11:29:55.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.981+0000 7f224679d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:55.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.981+0000 7f224679d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:55.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.981+0000 7f224679d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:29:55.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.981+0000 7f224679d640 1 -- 172.21.15.135:0/2968054451 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f224013cef0 con 0x7f224013f650 2024-03-29T11:29:55.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.984+0000 7f223cff9640 1 -- 172.21.15.135:0/2968054451 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f2234003bc0 con 0x7f224013f650 2024-03-29T11:29:55.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.985+0000 7f223cff9640 1 -- 172.21.15.135:0/2968054451 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f22340849a0 con 0x7f224013f650 2024-03-29T11:29:55.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.985+0000 7f223cff9640 10 client.4426.objecter ms_dispatch 0x7f2240000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:55.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.985+0000 7f223cff9640 3 client.4426.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:55.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.985+0000 7f223cff9640 3 client.4426.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:55.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.985+0000 7f223cff9640 20 client.4426.objecter dump_active .. 0 homeless 2024-03-29T11:29:55.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.985+0000 7f223ffff640 10 monclient: get_auth_request con 0x7f222c06dbe0 auth_method 0 2024-03-29T11:29:55.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.986+0000 7f223cff9640 10 client.4426.objecter ms_handle_connect 0x7f222c06dbe0 2024-03-29T11:29:55.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.989+0000 7f223cff9640 1 -- 172.21.15.135:0/2968054451 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f22340500b0 con 0x7f224013f650 2024-03-29T11:29:55.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.989+0000 7f223cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:55.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:55.989+0000 7f223cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:56.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:56.312+0000 7f224679d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:56.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:56.312+0000 7f224679d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:56.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:56.312+0000 7f224679d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:29:56.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:56.312+0000 7f224679d640 1 -- 172.21.15.135:0/2968054451 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22400b3680 con 0x7f224013f650 2024-03-29T11:29:56.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:56.317+0000 7f223cff9640 1 -- 172.21.15.135:0/2968054451 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f22340521b0 con 0x7f224013f650 2024-03-29T11:29:56.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:56.317+0000 7f223cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:56.310 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:56.317+0000 7f223cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:56.310 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:29:56.310 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:56.313 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:56.320+0000 7f224679d640 10 librados: watch_flush enter 2024-03-29T11:29:56.313 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:56.321+0000 7f224679d640 10 librados: watch_flush exit 2024-03-29T11:29:56.314 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:56.321+0000 7f224679d640 20 client.4426.objecter shutdown clearing up homeless session... 2024-03-29T11:29:56.314 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:56.321+0000 7f224679d640 10 client.4426.objecter successfully canceled tick 2024-03-29T11:29:56.314 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:56.321+0000 7f224679d640 1 -- 172.21.15.135:0/2968054451 >> v1:172.21.15.135:6816/3469875285 conn(0x7f222c06dbe0 legacy=0x7f222c0700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:56.314 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:56.321+0000 7f224679d640 10 monclient: shutdown 2024-03-29T11:29:56.314 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:56.321+0000 7f224679d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:56.314 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:56.321+0000 7f224679d640 1 -- 172.21.15.135:0/2968054451 >> v1:172.21.15.194:6789/0 conn(0x7f224013f650 legacy=0x7f2240143620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:56.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:56.322+0000 7f224679d640 1 -- 172.21.15.135:0/2968054451 shutdown_connections 2024-03-29T11:29:56.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:56.322+0000 7f224679d640 1 -- 172.21.15.135:0/2968054451 >> 172.21.15.135:0/2968054451 conn(0x7f2240066a00 msgr2=0x7f22400b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:56.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:56.323+0000 7f224679d640 1 -- 172.21.15.135:0/2968054451 shutdown_connections 2024-03-29T11:29:56.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:56.324+0000 7f224679d640 1 -- 172.21.15.135:0/2968054451 wait complete. 2024-03-29T11:29:56.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:29:56.324+0000 7f224679d640 1 librados: shutdown 2024-03-29T11:29:56.335 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:03.143 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:03.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.289+0000 7fb461495640 10 monclient: get_monmap_and_config 2024-03-29T11:30:03.281 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.289+0000 7fb461495640 10 monclient: build_initial_monmap 2024-03-29T11:30:03.281 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.289+0000 7fb461495640 10 monclient: monmap: 2024-03-29T11:30:03.281 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:30:03.281 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:03.281 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:30:03.289699+0000 2024-03-29T11:30:03.281 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:30:03.289699+0000 2024-03-29T11:30:03.281 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:03.281 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:03.281 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:30:03.281 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:30:03.281 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:30:03.282 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:03.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.290+0000 7fb461495640 1 Processor -- start 2024-03-29T11:30:03.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.290+0000 7fb461495640 1 -- start start 2024-03-29T11:30:03.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.290+0000 7fb461495640 10 monclient: init 2024-03-29T11:30:03.287 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.291+0000 7fb461495640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:03.287 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.291+0000 7fb461495640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:03.287 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.291+0000 7fb461495640 10 monclient(hunting): picked mon.noname-a con 0x7fb45c1457f0 addr v1:172.21.15.135:6789/0 2024-03-29T11:30:03.287 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.291+0000 7fb461495640 10 monclient(hunting): picked mon.noname-b con 0x7fb45c13f650 addr v1:172.21.15.135:6790/0 2024-03-29T11:30:03.287 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.291+0000 7fb461495640 10 monclient(hunting): picked mon.noname-c con 0x7fb45c0a9d70 addr v1:172.21.15.194:6789/0 2024-03-29T11:30:03.287 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.291+0000 7fb461495640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45c0aaef0 con 0x7fb45c1457f0 2024-03-29T11:30:03.287 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.291+0000 7fb461495640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45c145bd0 con 0x7fb45c0a9d70 2024-03-29T11:30:03.287 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.291+0000 7fb461495640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45c145eb0 con 0x7fb45c13f650 2024-03-29T11:30:03.287 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.291+0000 7fb461495640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:03.287 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.291+0000 7fb461495640 10 monclient(hunting): authenticate will time out at 1110.659474s 2024-03-29T11:30:03.287 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.291+0000 7fb45b7fe640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fb45c0a9d70 0x7fb45c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:43762/0 (socket says 172.21.15.135:43762) 2024-03-29T11:30:03.287 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.291+0000 7fb45b7fe640 1 -- 172.21.15.135:0/4288276581 learned_addr learned my addr 172.21.15.135:0/4288276581 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:30:03.287 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.292+0000 7fb459ffb640 1 -- 172.21.15.135:0/4288276581 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626374544 0 0) 0x7fb45c145eb0 con 0x7fb45c13f650 2024-03-29T11:30:03.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.292+0000 7fb459ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:03.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.292+0000 7fb459ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:03.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.292+0000 7fb459ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:03.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.292+0000 7fb459ffb640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:30:03.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.292+0000 7fb459ffb640 1 -- 172.21.15.135:0/4288276581 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb448003520 con 0x7fb45c13f650 2024-03-29T11:30:03.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.292+0000 7fb459ffb640 1 -- 172.21.15.135:0/4288276581 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608200647 0 0) 0x7fb45c145bd0 con 0x7fb45c0a9d70 2024-03-29T11:30:03.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.292+0000 7fb459ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:03.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.292+0000 7fb459ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:03.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.292+0000 7fb459ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:03.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.292+0000 7fb459ffb640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:30:03.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.292+0000 7fb459ffb640 1 -- 172.21.15.135:0/4288276581 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb45c145eb0 con 0x7fb45c0a9d70 2024-03-29T11:30:03.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.292+0000 7fb459ffb640 1 -- 172.21.15.135:0/4288276581 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 541404878 0 0) 0x7fb45c145eb0 con 0x7fb45c0a9d70 2024-03-29T11:30:03.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.292+0000 7fb459ffb640 1 -- 172.21.15.135:0/4288276581 >> v1:172.21.15.135:6790/0 conn(0x7fb45c13f650 legacy=0x7fb45c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:03.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.292+0000 7fb459ffb640 1 -- 172.21.15.135:0/4288276581 >> v1:172.21.15.135:6789/0 conn(0x7fb45c1457f0 legacy=0x7fb45c13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:03.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.294+0000 7fb459ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:30:03.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.294+0000 7fb459ffb640 1 monclient: found mon.noname-c 2024-03-29T11:30:03.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.294+0000 7fb459ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:03.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.294+0000 7fb459ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:30:03.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.294+0000 7fb459ffb640 1 -- 172.21.15.135:0/4288276581 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45c146340 con 0x7fb45c0a9d70 2024-03-29T11:30:03.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.294+0000 7fb459ffb640 10 monclient: _finish_auth 0 2024-03-29T11:30:03.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.294+0000 7fb459ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:30:03.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.294+0000 7fb459ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:03.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.294+0000 7fb461495640 5 monclient: authenticate success, global_id 4429 2024-03-29T11:30:03.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.294+0000 7fb461495640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:03.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.294+0000 7fb459ffb640 1 -- 172.21.15.135:0/4288276581 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fb4540019f0 con 0x7fb45c0a9d70 2024-03-29T11:30:03.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.294+0000 7fb459ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:03.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.294+0000 7fb459ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:03.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.294+0000 7fb459ffb640 10 monclient: dump: 2024-03-29T11:30:03.289 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:03.290 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:03.290 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:03.290 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:03.290 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:03.290 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:03.290 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:03.290 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:03.290 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:03.290 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:03.290 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.295+0000 7fb461495640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:03.290 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.295+0000 7fb459ffb640 1 -- 172.21.15.135:0/4288276581 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb454001c10 con 0x7fb45c0a9d70 2024-03-29T11:30:03.290 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.295+0000 7fb459ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:03.290 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.295+0000 7fb459ffb640 1 -- 172.21.15.135:0/4288276581 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fb454001fa0 con 0x7fb45c0a9d70 2024-03-29T11:30:03.290 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.295+0000 7fb459ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:03.291 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.295+0000 7fb459ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:03.291 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.295+0000 7fb459ffb640 10 monclient: dump: 2024-03-29T11:30:03.291 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:03.291 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:03.291 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:03.291 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:03.291 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:03.291 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:03.291 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:03.291 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:03.291 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:03.291 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:03.291 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.295+0000 7fb461495640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:03.291 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.295+0000 7fb461495640 4 set_mon_vals no callback set 2024-03-29T11:30:03.293 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.296+0000 7fb461495640 10 monclient: shutdown 2024-03-29T11:30:03.293 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.296+0000 7fb461495640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:03.293 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.296+0000 7fb461495640 1 -- 172.21.15.135:0/4288276581 >> v1:172.21.15.194:6789/0 conn(0x7fb45c0a9d70 legacy=0x7fb45c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:03.293 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.296+0000 7fb461495640 1 -- 172.21.15.135:0/4288276581 shutdown_connections 2024-03-29T11:30:03.293 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.296+0000 7fb461495640 1 -- 172.21.15.135:0/4288276581 >> 172.21.15.135:0/4288276581 conn(0x7fb45c066a00 msgr2=0x7fb45c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:03.293 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.296+0000 7fb461495640 1 -- 172.21.15.135:0/4288276581 shutdown_connections 2024-03-29T11:30:03.293 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.296+0000 7fb461495640 1 -- 172.21.15.135:0/4288276581 wait complete. 2024-03-29T11:30:03.294 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.297+0000 7fb461495640 10 monclient: build_initial_monmap 2024-03-29T11:30:03.294 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.297+0000 7fb461495640 10 monclient: monmap: 2024-03-29T11:30:03.294 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:30:03.294 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:03.294 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:30:03.294 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:30:03.294 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:03.294 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:03.294 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:30:03.294 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:30:03.294 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:30:03.294 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:03.294 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.297+0000 7fb461495640 1 librados: starting msgr at 2024-03-29T11:30:03.294 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.297+0000 7fb461495640 1 librados: starting objecter 2024-03-29T11:30:03.295 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.298+0000 7fb461495640 1 Processor -- start 2024-03-29T11:30:03.295 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.298+0000 7fb461495640 1 -- start start 2024-03-29T11:30:03.295 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.298+0000 7fb461495640 1 librados: setting wanted keys 2024-03-29T11:30:03.295 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.298+0000 7fb461495640 1 librados: calling monclient init 2024-03-29T11:30:03.295 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.298+0000 7fb461495640 10 monclient: init 2024-03-29T11:30:03.295 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.299+0000 7fb461495640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:03.295 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.299+0000 7fb461495640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:03.295 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.299+0000 7fb461495640 10 monclient(hunting): picked mon.noname-b con 0x7fb45c0a9d70 addr v1:172.21.15.194:6789/0 2024-03-29T11:30:03.295 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.299+0000 7fb461495640 10 monclient(hunting): picked mon.noname-a con 0x7fb45c13f650 addr v1:172.21.15.135:6789/0 2024-03-29T11:30:03.295 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.299+0000 7fb461495640 10 monclient(hunting): picked mon.noname-c con 0x7fb45c1457f0 addr v1:172.21.15.135:6790/0 2024-03-29T11:30:03.295 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.299+0000 7fb461495640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45c140540 con 0x7fb45c13f650 2024-03-29T11:30:03.295 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.299+0000 7fb461495640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45c140820 con 0x7fb45c0a9d70 2024-03-29T11:30:03.295 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.299+0000 7fb461495640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45c13b840 con 0x7fb45c1457f0 2024-03-29T11:30:03.295 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.299+0000 7fb461495640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:03.295 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.299+0000 7fb461495640 10 monclient(hunting): authenticate will time out at 1110.667496s 2024-03-29T11:30:03.296 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.300+0000 7fb45affd640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fb45c0a9d70 0x7fb45c13fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:44310/0 (socket says 172.21.15.135:44310) 2024-03-29T11:30:03.296 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.300+0000 7fb45affd640 1 -- 172.21.15.135:0/2965899418 learned_addr learned my addr 172.21.15.135:0/2965899418 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:30:03.296 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.301+0000 7fb4537fe640 10 client.?.objecter ms_handle_connect 0x7fb45c1457f0 2024-03-29T11:30:03.296 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.301+0000 7fb4537fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:03.296 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.301+0000 7fb4537fe640 1 -- 172.21.15.135:0/2965899418 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2080520832 0 0) 0x7fb45c13b840 con 0x7fb45c1457f0 2024-03-29T11:30:03.296 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.301+0000 7fb4537fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:03.296 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.301+0000 7fb4537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:03.296 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.301+0000 7fb4537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:03.296 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.301+0000 7fb4537fe640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:30:03.296 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.301+0000 7fb4537fe640 1 -- 172.21.15.135:0/2965899418 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb43c004eb0 con 0x7fb45c1457f0 2024-03-29T11:30:03.296 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.301+0000 7fb4537fe640 1 -- 172.21.15.135:0/2965899418 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3979161929 0 0) 0x7fb43c004eb0 con 0x7fb45c1457f0 2024-03-29T11:30:03.298 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb4537fe640 1 -- 172.21.15.135:0/2965899418 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb45c13b840 con 0x7fb45c1457f0 2024-03-29T11:30:03.298 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb4537fe640 1 -- 172.21.15.135:0/2965899418 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fb454001930 con 0x7fb45c1457f0 2024-03-29T11:30:03.298 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb4537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:03.298 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb4537fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:03.298 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb4537fe640 10 monclient(hunting): dump: 2024-03-29T11:30:03.298 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:03.298 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:03.298 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:03.298 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:03.299 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:03.299 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:03.299 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:03.299 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:03.299 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:03.299 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:03.299 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb4537fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:03.299 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb4537fe640 10 client.?.objecter ms_handle_connect 0x7fb45c0a9d70 2024-03-29T11:30:03.299 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb4537fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:03.299 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb4537fe640 1 -- 172.21.15.135:0/2965899418 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 283225834 0 0) 0x7fb45c13b840 con 0x7fb45c1457f0 2024-03-29T11:30:03.299 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb4537fe640 1 -- 172.21.15.135:0/2965899418 >> v1:172.21.15.194:6789/0 conn(0x7fb45c0a9d70 legacy=0x7fb45c13fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:03.299 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb4537fe640 1 -- 172.21.15.135:0/2965899418 >> v1:172.21.15.135:6789/0 conn(0x7fb45c13f650 legacy=0x7fb45c1d71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:03.299 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb4537fe640 10 monclient: _finish_hunting 0 2024-03-29T11:30:03.299 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb4537fe640 1 monclient: found mon.c 2024-03-29T11:30:03.300 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb4537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:03.300 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb4537fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:30:03.300 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb4537fe640 1 -- 172.21.15.135:0/2965899418 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45c1d7900 con 0x7fb45c1457f0 2024-03-29T11:30:03.300 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb4537fe640 10 monclient: _finish_auth 0 2024-03-29T11:30:03.300 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb4537fe640 10 monclient: _check_auth_tickets 2024-03-29T11:30:03.300 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb4537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:03.300 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb461495640 5 monclient: authenticate success, global_id 4439 2024-03-29T11:30:03.300 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb461495640 10 monclient: _renew_subs 2024-03-29T11:30:03.300 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb461495640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:30:03.300 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb461495640 1 -- 172.21.15.135:0/2965899418 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb45c1d4ae0 con 0x7fb45c1457f0 2024-03-29T11:30:03.300 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb461495640 10 client.4439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:03.300 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb461495640 10 monclient: _renew_subs 2024-03-29T11:30:03.300 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb461495640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:30:03.300 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb461495640 1 -- 172.21.15.135:0/2965899418 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb45c140cb0 con 0x7fb45c1457f0 2024-03-29T11:30:03.301 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.302+0000 7fb461495640 1 librados: init done 2024-03-29T11:30:03.301 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.303+0000 7fb4537fe640 1 -- 172.21.15.135:0/2965899418 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb454004e10 con 0x7fb45c1457f0 2024-03-29T11:30:03.301 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.303+0000 7fb4537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:03.301 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.303+0000 7fb4537fe640 1 -- 172.21.15.135:0/2965899418 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fb454005120 con 0x7fb45c1457f0 2024-03-29T11:30:03.301 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.303+0000 7fb4537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:03.301 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.303+0000 7fb4537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:03.301 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.303+0000 7fb4537fe640 10 monclient: dump: 2024-03-29T11:30:03.301 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:03.301 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:03.301 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:03.301 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:03.301 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:03.301 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:03.301 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:03.301 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:03.302 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:03.302 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:03.302 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.303+0000 7fb458ff9640 4 set_mon_vals no callback set 2024-03-29T11:30:03.302 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.304+0000 7fb4517fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:03.302 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.304+0000 7fb4517fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:03.302 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.304+0000 7fb4517fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:30:03.302 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.304+0000 7fb4517fa640 1 -- 172.21.15.135:0/2965899418 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb434005080 con 0x7fb45c1457f0 2024-03-29T11:30:03.302 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.305+0000 7fb4537fe640 1 -- 172.21.15.135:0/2965899418 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fb45401b0f0 con 0x7fb45c1457f0 2024-03-29T11:30:03.302 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.306+0000 7fb4537fe640 1 -- 172.21.15.135:0/2965899418 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7fb454084eb0 con 0x7fb45c1457f0 2024-03-29T11:30:03.302 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.306+0000 7fb4537fe640 10 client.4439.objecter ms_dispatch 0x7fb45c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:03.302 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.306+0000 7fb4537fe640 3 client.4439.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:03.302 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.306+0000 7fb4537fe640 3 client.4439.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:03.302 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.306+0000 7fb4537fe640 20 client.4439.objecter dump_active .. 0 homeless 2024-03-29T11:30:03.302 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.306+0000 7fb45affd640 10 monclient: get_auth_request con 0x7fb43c06dc60 auth_method 0 2024-03-29T11:30:03.302 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.307+0000 7fb4537fe640 10 client.4439.objecter ms_handle_connect 0x7fb43c06dc60 2024-03-29T11:30:03.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.314+0000 7fb4537fe640 1 -- 172.21.15.135:0/2965899418 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb4540507f0 con 0x7fb45c1457f0 2024-03-29T11:30:03.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.314+0000 7fb4537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:03.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.314+0000 7fb4537fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:03.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.626+0000 7fb4517fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:03.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.626+0000 7fb4517fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:03.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.626+0000 7fb4517fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:30:03.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.626+0000 7fb4517fa640 1 -- 172.21.15.135:0/2965899418 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb434005330 con 0x7fb45c1457f0 2024-03-29T11:30:03.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.627+0000 7fb4537fe640 1 -- 172.21.15.135:0/2965899418 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7fb454084560 con 0x7fb45c1457f0 2024-03-29T11:30:03.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.627+0000 7fb4537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:03.619 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.627+0000 7fb4537fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:03.619 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:30:03.619 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:03.620 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.629+0000 7fb4517fa640 10 librados: watch_flush enter 2024-03-29T11:30:03.621 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.629+0000 7fb4517fa640 10 librados: watch_flush exit 2024-03-29T11:30:03.621 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.629+0000 7fb4517fa640 20 client.4439.objecter shutdown clearing up homeless session... 2024-03-29T11:30:03.621 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.629+0000 7fb4517fa640 10 client.4439.objecter successfully canceled tick 2024-03-29T11:30:03.621 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.629+0000 7fb4517fa640 1 -- 172.21.15.135:0/2965899418 >> v1:172.21.15.135:6816/3469875285 conn(0x7fb43c06dc60 legacy=0x7fb43c070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:03.621 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.629+0000 7fb4517fa640 10 monclient: shutdown 2024-03-29T11:30:03.622 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.629+0000 7fb4517fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:03.622 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.629+0000 7fb4517fa640 1 -- 172.21.15.135:0/2965899418 >> v1:172.21.15.135:6790/0 conn(0x7fb45c1457f0 legacy=0x7fb45c1d43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:03.622 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.629+0000 7fb4517fa640 1 -- 172.21.15.135:0/2965899418 shutdown_connections 2024-03-29T11:30:03.622 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.629+0000 7fb4517fa640 1 -- 172.21.15.135:0/2965899418 >> 172.21.15.135:0/2965899418 conn(0x7fb45c066a00 msgr2=0x7fb45c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:03.622 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.629+0000 7fb4517fa640 1 -- 172.21.15.135:0/2965899418 shutdown_connections 2024-03-29T11:30:03.622 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.629+0000 7fb4517fa640 1 -- 172.21.15.135:0/2965899418 wait complete. 2024-03-29T11:30:03.622 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:03.630+0000 7fb4517fa640 1 librados: shutdown 2024-03-29T11:30:03.636 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:10.445 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:10.577 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.585+0000 7f15a3c57640 10 monclient: get_monmap_and_config 2024-03-29T11:30:10.577 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.585+0000 7f15a3c57640 10 monclient: build_initial_monmap 2024-03-29T11:30:10.577 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.585+0000 7f15a3c57640 10 monclient: monmap: 2024-03-29T11:30:10.577 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:30:10.578 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:10.578 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:30:10.586044+0000 2024-03-29T11:30:10.578 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:30:10.586044+0000 2024-03-29T11:30:10.578 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:10.578 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:10.578 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:30:10.578 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:30:10.578 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:30:10.578 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:10.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.586+0000 7f15a3c57640 1 Processor -- start 2024-03-29T11:30:10.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.586+0000 7f15a3c57640 1 -- start start 2024-03-29T11:30:10.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.586+0000 7f15a3c57640 10 monclient: init 2024-03-29T11:30:10.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.587+0000 7f15a3c57640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:10.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.587+0000 7f15a3c57640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:10.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.587+0000 7f15a3c57640 10 monclient(hunting): picked mon.noname-a con 0x7f159c1422b0 addr v1:172.21.15.135:6789/0 2024-03-29T11:30:10.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.587+0000 7f15a3c57640 10 monclient(hunting): picked mon.noname-b con 0x7f159c142690 addr v1:172.21.15.135:6790/0 2024-03-29T11:30:10.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.588+0000 7f15a3c57640 10 monclient(hunting): picked mon.noname-c con 0x7f159c0b17d0 addr v1:172.21.15.194:6789/0 2024-03-29T11:30:10.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.588+0000 7f15a3c57640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f159c142c80 con 0x7f159c1422b0 2024-03-29T11:30:10.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.588+0000 7f15a3c57640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f159c0aae40 con 0x7f159c0b17d0 2024-03-29T11:30:10.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.588+0000 7f15a21f4640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f159c0b17d0 0x7f159c0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:48954/0 (socket says 172.21.15.135:48954) 2024-03-29T11:30:10.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.588+0000 7f15a21f4640 1 -- 172.21.15.135:0/831797408 learned_addr learned my addr 172.21.15.135:0/831797408 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:30:10.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.588+0000 7f15a3c57640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f159c13fbd0 con 0x7f159c142690 2024-03-29T11:30:10.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.588+0000 7f15a3c57640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:10.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.588+0000 7f15a3c57640 10 monclient(hunting): authenticate will time out at 1117.956506s 2024-03-29T11:30:10.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.588+0000 7f15a09f1640 1 -- 172.21.15.135:0/831797408 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4012748292 0 0) 0x7f159c13fbd0 con 0x7f159c142690 2024-03-29T11:30:10.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.588+0000 7f15a09f1640 10 monclient(hunting): _negotiate 2024-03-29T11:30:10.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.588+0000 7f15a09f1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:10.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.588+0000 7f15a09f1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:10.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.588+0000 7f15a09f1640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:30:10.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.588+0000 7f15a09f1640 1 -- 172.21.15.135:0/831797408 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1588003520 con 0x7f159c142690 2024-03-29T11:30:10.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.589+0000 7f15a09f1640 1 -- 172.21.15.135:0/831797408 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1072254256 0 0) 0x7f159c0aae40 con 0x7f159c0b17d0 2024-03-29T11:30:10.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.589+0000 7f15a09f1640 10 monclient(hunting): _negotiate 2024-03-29T11:30:10.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.589+0000 7f15a09f1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:10.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.589+0000 7f15a09f1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:10.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.589+0000 7f15a09f1640 10 monclient(hunting): my global_id is 4435 2024-03-29T11:30:10.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.589+0000 7f15a09f1640 1 -- 172.21.15.135:0/831797408 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f159c13fbd0 con 0x7f159c0b17d0 2024-03-29T11:30:10.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.589+0000 7f15a09f1640 1 -- 172.21.15.135:0/831797408 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 485814862 0 0) 0x7f1588003520 con 0x7f159c142690 2024-03-29T11:30:10.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.589+0000 7f15a09f1640 1 -- 172.21.15.135:0/831797408 >> v1:172.21.15.194:6789/0 conn(0x7f159c0b17d0 legacy=0x7f159c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:10.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.589+0000 7f15a09f1640 1 -- 172.21.15.135:0/831797408 >> v1:172.21.15.135:6789/0 conn(0x7f159c1422b0 legacy=0x7f159c13f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:10.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.590+0000 7f15a09f1640 10 monclient: _finish_hunting 0 2024-03-29T11:30:10.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.590+0000 7f15a09f1640 1 monclient: found mon.noname-b 2024-03-29T11:30:10.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.590+0000 7f15a09f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:10.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.590+0000 7f15a09f1640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:30:10.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.590+0000 7f15a09f1640 1 -- 172.21.15.135:0/831797408 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f159c151940 con 0x7f159c142690 2024-03-29T11:30:10.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.590+0000 7f15a09f1640 10 monclient: _finish_auth 0 2024-03-29T11:30:10.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.590+0000 7f15a09f1640 10 monclient: _check_auth_tickets 2024-03-29T11:30:10.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.590+0000 7f15a09f1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:10.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.590+0000 7f15a09f1640 1 -- 172.21.15.135:0/831797408 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f158c001690 con 0x7f159c142690 2024-03-29T11:30:10.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.590+0000 7f15a09f1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:10.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.590+0000 7f15a09f1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:10.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.590+0000 7f15a09f1640 10 monclient: dump: 2024-03-29T11:30:10.588 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:10.588 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:10.588 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:10.588 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:10.588 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:10.588 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:10.588 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:10.588 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:10.588 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:10.588 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:10.589 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.590+0000 7f15a3c57640 5 monclient: authenticate success, global_id 4442 2024-03-29T11:30:10.589 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.590+0000 7f15a3c57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:10.589 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.590+0000 7f15a09f1640 1 -- 172.21.15.135:0/831797408 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f158c0017f0 con 0x7f159c142690 2024-03-29T11:30:10.589 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.590+0000 7f15a09f1640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:10.589 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.590+0000 7f15a3c57640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:10.589 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.590+0000 7f15a3c57640 4 set_mon_vals no callback set 2024-03-29T11:30:10.589 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.590+0000 7f15a09f1640 1 -- 172.21.15.135:0/831797408 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f158c001b80 con 0x7f159c142690 2024-03-29T11:30:10.589 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.590+0000 7f15a09f1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:10.589 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.590+0000 7f15a09f1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:10.589 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.590+0000 7f15a09f1640 10 monclient: dump: 2024-03-29T11:30:10.589 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:10.589 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:10.589 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:10.589 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:10.589 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:10.589 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:10.590 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:10.590 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:10.590 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:10.590 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:10.590 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.591+0000 7f15a3c57640 10 monclient: shutdown 2024-03-29T11:30:10.590 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.591+0000 7f15a3c57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:10.590 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.591+0000 7f15a3c57640 1 -- 172.21.15.135:0/831797408 >> v1:172.21.15.135:6790/0 conn(0x7f159c142690 legacy=0x7f159c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:10.590 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.591+0000 7f15a3c57640 1 -- 172.21.15.135:0/831797408 shutdown_connections 2024-03-29T11:30:10.590 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.591+0000 7f15a3c57640 1 -- 172.21.15.135:0/831797408 >> 172.21.15.135:0/831797408 conn(0x7f159c066a00 msgr2=0x7f159c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:10.590 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.592+0000 7f15a3c57640 1 -- 172.21.15.135:0/831797408 shutdown_connections 2024-03-29T11:30:10.590 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.592+0000 7f15a3c57640 1 -- 172.21.15.135:0/831797408 wait complete. 2024-03-29T11:30:10.590 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.593+0000 7f15a3c57640 10 monclient: build_initial_monmap 2024-03-29T11:30:10.590 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.593+0000 7f15a3c57640 10 monclient: monmap: 2024-03-29T11:30:10.590 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:30:10.591 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:10.591 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:30:10.591 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:30:10.591 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:10.591 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:10.591 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:30:10.591 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:30:10.591 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:30:10.591 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:10.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.593+0000 7f15a3c57640 1 librados: starting msgr at 2024-03-29T11:30:10.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.593+0000 7f15a3c57640 1 librados: starting objecter 2024-03-29T11:30:10.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.593+0000 7f15a3c57640 1 Processor -- start 2024-03-29T11:30:10.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.593+0000 7f15a3c57640 1 -- start start 2024-03-29T11:30:10.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.593+0000 7f15a3c57640 1 librados: setting wanted keys 2024-03-29T11:30:10.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.593+0000 7f15a3c57640 1 librados: calling monclient init 2024-03-29T11:30:10.592 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.593+0000 7f15a3c57640 10 monclient: init 2024-03-29T11:30:10.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.594+0000 7f15a3c57640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:10.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.594+0000 7f15a3c57640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:30:10.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.594+0000 7f15a3c57640 10 monclient(hunting): picked mon.noname-c con 0x7f159c0b17d0 addr v1:172.21.15.135:6790/0 2024-03-29T11:30:10.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.594+0000 7f15a3c57640 10 monclient(hunting): picked mon.noname-b con 0x7f159c1422b0 addr v1:172.21.15.194:6789/0 2024-03-29T11:30:10.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.594+0000 7f15a3c57640 10 monclient(hunting): picked mon.noname-a con 0x7f159c142690 addr v1:172.21.15.135:6789/0 2024-03-29T11:30:10.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.594+0000 7f15a3c57640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f159c0acaf0 con 0x7f159c142690 2024-03-29T11:30:10.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.594+0000 7f15a3c57640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f159c0acdd0 con 0x7f159c1422b0 2024-03-29T11:30:10.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.594+0000 7f15a3c57640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f159c13c750 con 0x7f159c0b17d0 2024-03-29T11:30:10.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.594+0000 7f15a3c57640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:10.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.594+0000 7f15a3c57640 10 monclient(hunting): authenticate will time out at 1117.962163s 2024-03-29T11:30:10.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.594+0000 7f15a19f3640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f159c0b17d0 0x7f159c14d9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:60376/0 (socket says 172.21.15.135:60376) 2024-03-29T11:30:10.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.594+0000 7f15a19f3640 1 -- 172.21.15.135:0/3591420754 learned_addr learned my addr 172.21.15.135:0/3591420754 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:30:10.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.594+0000 7f15927fc640 10 client.?.objecter ms_handle_connect 0x7f159c0b17d0 2024-03-29T11:30:10.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.594+0000 7f15927fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:10.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.594+0000 7f15927fc640 1 -- 172.21.15.135:0/3591420754 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2953710467 0 0) 0x7f159c13c750 con 0x7f159c0b17d0 2024-03-29T11:30:10.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.594+0000 7f15927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:10.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.594+0000 7f15927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:10.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.594+0000 7f15927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:10.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.595+0000 7f15927fc640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:30:10.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.595+0000 7f15927fc640 1 -- 172.21.15.135:0/3591420754 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1580004eb0 con 0x7f159c0b17d0 2024-03-29T11:30:10.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.595+0000 7f15a19f3640 1 -- 172.21.15.135:0/3591420754 >> v1:172.21.15.135:6790/0 conn(0x7f159c0b17d0 legacy=0x7f159c14d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-29T11:30:10.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.595+0000 7f15a19f3640 1 -- 172.21.15.135:0/3591420754 >> v1:172.21.15.135:6790/0 conn(0x7f159c0b17d0 legacy=0x7f159c14d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:30:10.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.595+0000 7f15a19f3640 1 --1- 172.21.15.135:0/3591420754 >> v1:172.21.15.135:6790/0 conn(0x7f159c0b17d0 0x7f159c14d9d0 :-1 s=OPENED pgs=561566 cs=1 l=1).handle_message read tag failed 2024-03-29T11:30:10.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.595+0000 7f15a19f3640 1 --1- 172.21.15.135:0/3591420754 >> v1:172.21.15.135:6790/0 conn(0x7f159c0b17d0 0x7f159c14d9d0 :-1 s=OPENED pgs=561566 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:30:10.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.595+0000 7f15927fc640 10 client.?.objecter ms_handle_connect 0x7f159c1422b0 2024-03-29T11:30:10.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.595+0000 7f15927fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:10.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.595+0000 7f15927fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.135:6790/0 2024-03-29T11:30:10.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.595+0000 7f15927fc640 1 -- 172.21.15.135:0/3591420754 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 885119081 0 0) 0x7f159c0acdd0 con 0x7f159c1422b0 2024-03-29T11:30:10.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.595+0000 7f15927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:10.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.596+0000 7f15927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:10.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.596+0000 7f15927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:10.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.596+0000 7f15927fc640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:30:10.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.596+0000 7f15927fc640 1 -- 172.21.15.135:0/3591420754 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f159c13c750 con 0x7f159c1422b0 2024-03-29T11:30:10.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.597+0000 7f15927fc640 1 -- 172.21.15.135:0/3591420754 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 683763642 0 0) 0x7f159c13c750 con 0x7f159c1422b0 2024-03-29T11:30:10.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.597+0000 7f15927fc640 1 -- 172.21.15.135:0/3591420754 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f159c0acdd0 con 0x7f159c1422b0 2024-03-29T11:30:10.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.597+0000 7f15927fc640 10 client.?.objecter ms_handle_connect 0x7f159c142690 2024-03-29T11:30:10.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.597+0000 7f15927fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:10.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.597+0000 7f15927fc640 1 -- 172.21.15.135:0/3591420754 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f158c001b00 con 0x7f159c1422b0 2024-03-29T11:30:10.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.597+0000 7f15927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:10.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.597+0000 7f15927fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:10.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.597+0000 7f15927fc640 10 monclient(hunting): dump: 2024-03-29T11:30:10.602 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:10.602 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:10.602 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:10.602 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:10.602 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:10.602 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:10.602 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:10.602 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:10.602 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:10.603 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:10.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.598+0000 7f15927fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:10.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.598+0000 7f15927fc640 1 -- 172.21.15.135:0/3591420754 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1333242378 0 0) 0x7f159c0acaf0 con 0x7f159c142690 2024-03-29T11:30:10.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.598+0000 7f15927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:10.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.598+0000 7f15927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:10.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.598+0000 7f15927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:10.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.598+0000 7f15927fc640 10 monclient(hunting): my global_id is 4428 2024-03-29T11:30:10.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.598+0000 7f15927fc640 1 -- 172.21.15.135:0/3591420754 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f159c13c750 con 0x7f159c142690 2024-03-29T11:30:10.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.598+0000 7f15927fc640 1 -- 172.21.15.135:0/3591420754 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2332760298 0 0) 0x7f159c13c750 con 0x7f159c142690 2024-03-29T11:30:10.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.598+0000 7f15927fc640 1 -- 172.21.15.135:0/3591420754 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f159c0acaf0 con 0x7f159c142690 2024-03-29T11:30:10.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.598+0000 7f15927fc640 1 -- 172.21.15.135:0/3591420754 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f1594001cb0 con 0x7f159c142690 2024-03-29T11:30:10.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.598+0000 7f15927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:10.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.598+0000 7f15927fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:10.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.598+0000 7f15927fc640 10 monclient(hunting): dump: 2024-03-29T11:30:10.604 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:10.604 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:10.604 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:10.605 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:10.605 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:10.605 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:10.605 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:10.605 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:10.605 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:10.605 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:10.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.598+0000 7f15927fc640 1 -- 172.21.15.135:0/3591420754 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 532815994 0 0) 0x7f159c0acaf0 con 0x7f159c142690 2024-03-29T11:30:10.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.598+0000 7f15927fc640 1 -- 172.21.15.135:0/3591420754 >> v1:172.21.15.135:6790/0 conn(0x7f159c0b17d0 legacy=0x7f159c14d9d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:30:10.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.599+0000 7f15927fc640 1 -- 172.21.15.135:0/3591420754 >> v1:172.21.15.194:6789/0 conn(0x7f159c1422b0 legacy=0x7f159c12ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:10.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.599+0000 7f15927fc640 10 monclient: _finish_hunting 0 2024-03-29T11:30:10.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.599+0000 7f15927fc640 1 monclient: found mon.a 2024-03-29T11:30:10.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.599+0000 7f15927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:10.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.599+0000 7f15927fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:30:10.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.599+0000 7f15927fc640 1 -- 172.21.15.135:0/3591420754 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f159c132540 con 0x7f159c142690 2024-03-29T11:30:10.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.599+0000 7f15927fc640 10 monclient: _finish_auth 0 2024-03-29T11:30:10.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.599+0000 7f15927fc640 10 monclient: _check_auth_tickets 2024-03-29T11:30:10.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.599+0000 7f15927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:10.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.599+0000 7f15927fc640 1 -- 172.21.15.135:0/3591420754 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1594004b60 con 0x7f159c142690 2024-03-29T11:30:10.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.599+0000 7f15a3c57640 5 monclient: authenticate success, global_id 4428 2024-03-29T11:30:10.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.599+0000 7f15a3c57640 10 monclient: _renew_subs 2024-03-29T11:30:10.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.599+0000 7f15a3c57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:30:10.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.599+0000 7f15a3c57640 1 -- 172.21.15.135:0/3591420754 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f159c1320b0 con 0x7f159c142690 2024-03-29T11:30:10.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.599+0000 7f15a3c57640 10 client.4428.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:10.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.599+0000 7f15a3c57640 10 monclient: _renew_subs 2024-03-29T11:30:10.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.599+0000 7f15a3c57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:30:10.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.599+0000 7f15a3c57640 1 -- 172.21.15.135:0/3591420754 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f159c0ab990 con 0x7f159c142690 2024-03-29T11:30:10.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.599+0000 7f15a3c57640 1 librados: init done 2024-03-29T11:30:10.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.600+0000 7f15927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:10.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.600+0000 7f15927fc640 1 -- 172.21.15.135:0/3591420754 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f1594004e70 con 0x7f159c142690 2024-03-29T11:30:10.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.600+0000 7f15927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:10.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.600+0000 7f15927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:10.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.600+0000 7f15927fc640 10 monclient: dump: 2024-03-29T11:30:10.607 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:10.607 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:10.607 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:10.607 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:10.607 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:10.607 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:10.608 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:10.608 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:10.608 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:10.608 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:10.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.600+0000 7f1593fff640 4 set_mon_vals no callback set 2024-03-29T11:30:10.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.601+0000 7f15a3c57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:10.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.601+0000 7f15a3c57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:10.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.601+0000 7f15a3c57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:30:10.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.601+0000 7f15a3c57640 1 -- 172.21.15.135:0/3591420754 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1584005080 con 0x7f159c142690 2024-03-29T11:30:10.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.602+0000 7f15927fc640 1 -- 172.21.15.135:0/3591420754 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f159401ae40 con 0x7f159c142690 2024-03-29T11:30:10.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.603+0000 7f15927fc640 1 -- 172.21.15.135:0/3591420754 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f1594084c40 con 0x7f159c142690 2024-03-29T11:30:10.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.603+0000 7f15927fc640 10 client.4428.objecter ms_dispatch 0x7f159c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:10.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.603+0000 7f15927fc640 3 client.4428.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:10.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.603+0000 7f15927fc640 3 client.4428.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:10.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.603+0000 7f15927fc640 20 client.4428.objecter dump_active .. 0 homeless 2024-03-29T11:30:10.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.603+0000 7f15a19f3640 10 monclient: get_auth_request con 0x7f158006e020 auth_method 0 2024-03-29T11:30:10.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.603+0000 7f15927fc640 10 client.4428.objecter ms_handle_connect 0x7f158006e020 2024-03-29T11:30:10.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.609+0000 7f15927fc640 1 -- 172.21.15.135:0/3591420754 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1594050520 con 0x7f159c142690 2024-03-29T11:30:10.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.610+0000 7f15927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:10.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.610+0000 7f15927fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:10.919 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.926+0000 7f15a3c57640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:10.919 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.926+0000 7f15a3c57640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:10.919 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.926+0000 7f15a3c57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:30:10.919 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.926+0000 7f15a3c57640 1 -- 172.21.15.135:0/3591420754 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1584005330 con 0x7f159c142690 2024-03-29T11:30:10.919 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.927+0000 7f15927fc640 1 -- 172.21.15.135:0/3591420754 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f1594052450 con 0x7f159c142690 2024-03-29T11:30:10.919 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.927+0000 7f15927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:10.919 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.927+0000 7f15927fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:10.919 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:30:10.920 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:10.924 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.930+0000 7f15a3c57640 10 librados: watch_flush enter 2024-03-29T11:30:10.924 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.930+0000 7f15a3c57640 10 librados: watch_flush exit 2024-03-29T11:30:10.924 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.930+0000 7f15a3c57640 20 client.4428.objecter shutdown clearing up homeless session... 2024-03-29T11:30:10.924 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.930+0000 7f15a3c57640 10 client.4428.objecter successfully canceled tick 2024-03-29T11:30:10.924 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.930+0000 7f15a3c57640 1 -- 172.21.15.135:0/3591420754 >> v1:172.21.15.135:6816/3469875285 conn(0x7f158006e020 legacy=0x7f15800704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:10.925 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.930+0000 7f15a3c57640 10 monclient: shutdown 2024-03-29T11:30:10.925 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.930+0000 7f15a3c57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:10.925 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.930+0000 7f15a3c57640 1 -- 172.21.15.135:0/3591420754 >> v1:172.21.15.135:6789/0 conn(0x7f159c142690 legacy=0x7f159c130680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:10.925 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.930+0000 7f15a3c57640 1 -- 172.21.15.135:0/3591420754 shutdown_connections 2024-03-29T11:30:10.925 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.930+0000 7f15a3c57640 1 -- 172.21.15.135:0/3591420754 >> 172.21.15.135:0/3591420754 conn(0x7f159c066a00 msgr2=0x7f159c13db20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:10.925 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.931+0000 7f15a3c57640 1 -- 172.21.15.135:0/3591420754 shutdown_connections 2024-03-29T11:30:10.926 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.931+0000 7f15a3c57640 1 -- 172.21.15.135:0/3591420754 wait complete. 2024-03-29T11:30:10.926 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:10.931+0000 7f15a3c57640 1 librados: shutdown 2024-03-29T11:30:10.938 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:17.745 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:17.895 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.902+0000 7fda22c33640 10 monclient: get_monmap_and_config 2024-03-29T11:30:17.895 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.902+0000 7fda22c33640 10 monclient: build_initial_monmap 2024-03-29T11:30:17.895 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.902+0000 7fda22c33640 10 monclient: monmap: 2024-03-29T11:30:17.895 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:30:17.895 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:17.895 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:30:17.903499+0000 2024-03-29T11:30:17.895 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:30:17.903499+0000 2024-03-29T11:30:17.895 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:17.895 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:17.895 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:30:17.895 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:30:17.895 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:30:17.896 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:17.896 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.905+0000 7fda22c33640 1 Processor -- start 2024-03-29T11:30:17.897 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.905+0000 7fda22c33640 1 -- start start 2024-03-29T11:30:17.897 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.905+0000 7fda22c33640 10 monclient: init 2024-03-29T11:30:17.897 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.905+0000 7fda22c33640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:17.898 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.906+0000 7fda22c33640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:17.898 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.906+0000 7fda22c33640 10 monclient(hunting): picked mon.noname-c con 0x7fda1c1457f0 addr v1:172.21.15.194:6789/0 2024-03-29T11:30:17.900 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.907+0000 7fda22c33640 10 monclient(hunting): picked mon.noname-a con 0x7fda1c13f650 addr v1:172.21.15.135:6789/0 2024-03-29T11:30:17.900 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.907+0000 7fda22c33640 10 monclient(hunting): picked mon.noname-b con 0x7fda1c0a9d70 addr v1:172.21.15.135:6790/0 2024-03-29T11:30:17.900 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.907+0000 7fda22c33640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda1c0aaef0 con 0x7fda1c13f650 2024-03-29T11:30:17.900 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.907+0000 7fda22c33640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda1c145bd0 con 0x7fda1c1457f0 2024-03-29T11:30:17.900 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.907+0000 7fda22c33640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda1c145eb0 con 0x7fda1c0a9d70 2024-03-29T11:30:17.900 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.907+0000 7fda22c33640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:17.900 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.907+0000 7fda22c33640 10 monclient(hunting): authenticate will time out at 1125.275925s 2024-03-29T11:30:17.901 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.909+0000 7fda211d0640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7fda1c0a9d70 0x7fda1c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:58024/0 (socket says 172.21.15.135:58024) 2024-03-29T11:30:17.901 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.909+0000 7fda211d0640 1 -- 172.21.15.135:0/528248413 learned_addr learned my addr 172.21.15.135:0/528248413 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:30:17.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.909+0000 7fda1b7fe640 1 -- 172.21.15.135:0/528248413 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384325814 0 0) 0x7fda1c145eb0 con 0x7fda1c0a9d70 2024-03-29T11:30:17.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.909+0000 7fda1b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:17.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.909+0000 7fda1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:17.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.909+0000 7fda1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:17.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.909+0000 7fda1b7fe640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:30:17.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.909+0000 7fda1b7fe640 1 -- 172.21.15.135:0/528248413 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda04003520 con 0x7fda1c0a9d70 2024-03-29T11:30:17.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.909+0000 7fda1b7fe640 1 -- 172.21.15.135:0/528248413 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1900117031 0 0) 0x7fda1c0aaef0 con 0x7fda1c13f650 2024-03-29T11:30:17.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.909+0000 7fda1b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:17.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.909+0000 7fda1b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:17.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.909+0000 7fda1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:17.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.909+0000 7fda1b7fe640 10 monclient(hunting): my global_id is 4431 2024-03-29T11:30:17.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.909+0000 7fda1b7fe640 1 -- 172.21.15.135:0/528248413 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda1c145eb0 con 0x7fda1c13f650 2024-03-29T11:30:17.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.910+0000 7fda1b7fe640 1 -- 172.21.15.135:0/528248413 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2521585785 0 0) 0x7fda1c145eb0 con 0x7fda1c13f650 2024-03-29T11:30:17.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.910+0000 7fda1b7fe640 1 -- 172.21.15.135:0/528248413 >> v1:172.21.15.135:6790/0 conn(0x7fda1c0a9d70 legacy=0x7fda1c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:17.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.910+0000 7fda1b7fe640 1 -- 172.21.15.135:0/528248413 >> v1:172.21.15.194:6789/0 conn(0x7fda1c1457f0 legacy=0x7fda1c13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:17.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.911+0000 7fda1b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:30:17.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.911+0000 7fda1b7fe640 1 monclient: found mon.noname-a 2024-03-29T11:30:17.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.911+0000 7fda1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:17.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.911+0000 7fda1b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.135:6789/0 2024-03-29T11:30:17.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.911+0000 7fda1b7fe640 1 -- 172.21.15.135:0/528248413 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda1c146340 con 0x7fda1c13f650 2024-03-29T11:30:17.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.911+0000 7fda1b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:30:17.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.911+0000 7fda1b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:30:17.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.911+0000 7fda1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:17.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.911+0000 7fda1b7fe640 1 -- 172.21.15.135:0/528248413 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fda140019f0 con 0x7fda1c13f650 2024-03-29T11:30:17.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.911+0000 7fda22c33640 5 monclient: authenticate success, global_id 4431 2024-03-29T11:30:17.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.911+0000 7fda22c33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:17.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.911+0000 7fda1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:17.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.911+0000 7fda1b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:17.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.911+0000 7fda1b7fe640 10 monclient: dump: 2024-03-29T11:30:17.905 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:17.905 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:17.905 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:17.905 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:17.905 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:17.905 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:17.906 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:17.906 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:17.906 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:17.906 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:17.906 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.911+0000 7fda22c33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:17.907 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.912+0000 7fda1b7fe640 1 -- 172.21.15.135:0/528248413 <== mon.0 v1:172.21.15.135:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda14001c10 con 0x7fda1c13f650 2024-03-29T11:30:17.907 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.912+0000 7fda1b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:17.907 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.912+0000 7fda1b7fe640 1 -- 172.21.15.135:0/528248413 <== mon.0 v1:172.21.15.135:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fda14001fa0 con 0x7fda1c13f650 2024-03-29T11:30:17.907 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.912+0000 7fda1b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:17.907 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.912+0000 7fda1b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:17.907 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.912+0000 7fda1b7fe640 10 monclient: dump: 2024-03-29T11:30:17.907 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:17.907 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:17.907 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:17.908 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:17.908 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:17.908 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:17.908 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:17.908 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:17.908 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:17.908 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:17.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.912+0000 7fda22c33640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:17.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.912+0000 7fda22c33640 4 set_mon_vals no callback set 2024-03-29T11:30:17.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.913+0000 7fda22c33640 10 monclient: shutdown 2024-03-29T11:30:17.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.913+0000 7fda22c33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:17.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.913+0000 7fda22c33640 1 -- 172.21.15.135:0/528248413 >> v1:172.21.15.135:6789/0 conn(0x7fda1c13f650 legacy=0x7fda1c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:17.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.913+0000 7fda22c33640 1 -- 172.21.15.135:0/528248413 shutdown_connections 2024-03-29T11:30:17.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.913+0000 7fda22c33640 1 -- 172.21.15.135:0/528248413 >> 172.21.15.135:0/528248413 conn(0x7fda1c066a00 msgr2=0x7fda1c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:17.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.914+0000 7fda22c33640 1 -- 172.21.15.135:0/528248413 shutdown_connections 2024-03-29T11:30:17.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.914+0000 7fda22c33640 1 -- 172.21.15.135:0/528248413 wait complete. 2024-03-29T11:30:17.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.915+0000 7fda22c33640 10 monclient: build_initial_monmap 2024-03-29T11:30:17.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.915+0000 7fda22c33640 10 monclient: monmap: 2024-03-29T11:30:17.909 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:30:17.909 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:17.909 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:30:17.909 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:30:17.910 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:17.910 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:17.910 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:30:17.910 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:30:17.910 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:30:17.910 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:17.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.916+0000 7fda22c33640 1 librados: starting msgr at 2024-03-29T11:30:17.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.916+0000 7fda22c33640 1 librados: starting objecter 2024-03-29T11:30:17.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.916+0000 7fda22c33640 1 Processor -- start 2024-03-29T11:30:17.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.917+0000 7fda22c33640 1 -- start start 2024-03-29T11:30:17.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.917+0000 7fda22c33640 1 librados: setting wanted keys 2024-03-29T11:30:17.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.917+0000 7fda22c33640 1 librados: calling monclient init 2024-03-29T11:30:17.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.917+0000 7fda22c33640 10 monclient: init 2024-03-29T11:30:17.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.919+0000 7fda22c33640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:17.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.919+0000 7fda22c33640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:30:17.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.919+0000 7fda22c33640 10 monclient(hunting): picked mon.noname-c con 0x7fda1c0a9d70 addr v1:172.21.15.135:6790/0 2024-03-29T11:30:17.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.919+0000 7fda22c33640 10 monclient(hunting): picked mon.noname-b con 0x7fda1c1457f0 addr v1:172.21.15.194:6789/0 2024-03-29T11:30:17.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.920+0000 7fda22c33640 10 monclient(hunting): picked mon.noname-a con 0x7fda1c139da0 addr v1:172.21.15.135:6789/0 2024-03-29T11:30:17.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.920+0000 7fda22c33640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda1c13ba10 con 0x7fda1c139da0 2024-03-29T11:30:17.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.920+0000 7fda22c33640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda1c13acc0 con 0x7fda1c1457f0 2024-03-29T11:30:17.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.920+0000 7fda22c33640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda1c1dbbd0 con 0x7fda1c0a9d70 2024-03-29T11:30:17.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.920+0000 7fda22c33640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:17.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.920+0000 7fda22c33640 10 monclient(hunting): authenticate will time out at 1125.288947s 2024-03-29T11:30:17.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.919+0000 7fda209cf640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7fda1c0a9d70 0x7fda1c139690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:46310/0 (socket says 172.21.15.135:46310) 2024-03-29T11:30:17.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.920+0000 7fda209cf640 1 -- 172.21.15.135:0/3801578744 learned_addr learned my addr 172.21.15.135:0/3801578744 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:30:17.920 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.924+0000 7fda197fa640 10 client.?.objecter ms_handle_connect 0x7fda1c0a9d70 2024-03-29T11:30:17.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.924+0000 7fda197fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:17.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.924+0000 7fda197fa640 1 -- 172.21.15.135:0/3801578744 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581076770 0 0) 0x7fda1c1dbbd0 con 0x7fda1c0a9d70 2024-03-29T11:30:17.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.924+0000 7fda197fa640 10 monclient(hunting): _negotiate 2024-03-29T11:30:17.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.924+0000 7fda197fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:17.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.924+0000 7fda197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:17.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.924+0000 7fda197fa640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:30:17.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.924+0000 7fda197fa640 1 -- 172.21.15.135:0/3801578744 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda08004eb0 con 0x7fda1c0a9d70 2024-03-29T11:30:17.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.925+0000 7fda197fa640 1 -- 172.21.15.135:0/3801578744 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1811475943 0 0) 0x7fda08004eb0 con 0x7fda1c0a9d70 2024-03-29T11:30:17.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.925+0000 7fda197fa640 1 -- 172.21.15.135:0/3801578744 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda1c1dbbd0 con 0x7fda1c0a9d70 2024-03-29T11:30:17.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.925+0000 7fda197fa640 1 -- 172.21.15.135:0/3801578744 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fda100017d0 con 0x7fda1c0a9d70 2024-03-29T11:30:17.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.925+0000 7fda197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:17.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.925+0000 7fda197fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:17.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.925+0000 7fda197fa640 10 monclient(hunting): dump: 2024-03-29T11:30:17.922 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:17.922 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:17.922 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:17.922 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:17.922 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:17.922 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:17.922 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:17.922 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:17.922 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:17.922 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:17.922 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.925+0000 7fda197fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:17.922 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.926+0000 7fda197fa640 10 client.?.objecter ms_handle_connect 0x7fda1c139da0 2024-03-29T11:30:17.922 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.926+0000 7fda197fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:17.922 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.926+0000 7fda197fa640 1 -- 172.21.15.135:0/3801578744 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3593018792 0 0) 0x7fda1c1dbbd0 con 0x7fda1c0a9d70 2024-03-29T11:30:17.923 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.926+0000 7fda197fa640 1 -- 172.21.15.135:0/3801578744 >> v1:172.21.15.194:6789/0 conn(0x7fda1c1457f0 legacy=0x7fda1c1dadb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:30:17.923 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.926+0000 7fda197fa640 1 -- 172.21.15.135:0/3801578744 >> v1:172.21.15.135:6789/0 conn(0x7fda1c139da0 legacy=0x7fda1c1db4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:17.923 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.926+0000 7fda197fa640 10 monclient: _finish_hunting 0 2024-03-29T11:30:17.923 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.926+0000 7fda197fa640 1 monclient: found mon.c 2024-03-29T11:30:17.923 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.926+0000 7fda197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:17.923 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.926+0000 7fda197fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:30:17.923 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.927+0000 7fda197fa640 1 -- 172.21.15.135:0/3801578744 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda1c13af30 con 0x7fda1c0a9d70 2024-03-29T11:30:17.923 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.927+0000 7fda197fa640 10 monclient: _finish_auth 0 2024-03-29T11:30:17.923 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.927+0000 7fda197fa640 10 monclient: _check_auth_tickets 2024-03-29T11:30:17.923 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.927+0000 7fda197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:17.923 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.927+0000 7fda197fa640 1 -- 172.21.15.135:0/3801578744 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda10001930 con 0x7fda1c0a9d70 2024-03-29T11:30:17.923 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.927+0000 7fda197fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:17.923 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.927+0000 7fda1a7fc640 4 set_mon_vals no callback set 2024-03-29T11:30:17.923 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.927+0000 7fda197fa640 1 -- 172.21.15.135:0/3801578744 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fda10004a40 con 0x7fda1c0a9d70 2024-03-29T11:30:17.924 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.927+0000 7fda197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:17.924 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.927+0000 7fda197fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:17.924 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.927+0000 7fda197fa640 10 monclient: dump: 2024-03-29T11:30:17.924 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:17.924 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:17.924 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:17.924 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:17.924 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:17.924 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:17.924 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:17.924 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:17.924 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:17.924 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:17.924 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.927+0000 7fda22c33640 5 monclient: authenticate success, global_id 4451 2024-03-29T11:30:17.924 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.927+0000 7fda22c33640 10 monclient: _renew_subs 2024-03-29T11:30:17.925 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.927+0000 7fda22c33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:30:17.925 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.927+0000 7fda22c33640 1 -- 172.21.15.135:0/3801578744 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda1c13b210 con 0x7fda1c0a9d70 2024-03-29T11:30:17.925 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.927+0000 7fda22c33640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:17.925 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.927+0000 7fda22c33640 10 monclient: _renew_subs 2024-03-29T11:30:17.925 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.927+0000 7fda22c33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:30:17.925 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.927+0000 7fda22c33640 1 -- 172.21.15.135:0/3801578744 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda1c0ab600 con 0x7fda1c0a9d70 2024-03-29T11:30:17.925 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.927+0000 7fda22c33640 1 librados: init done 2024-03-29T11:30:17.927 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.930+0000 7fda22c33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:17.927 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.930+0000 7fda22c33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:17.927 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.930+0000 7fda22c33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:30:17.927 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.930+0000 7fda22c33640 1 -- 172.21.15.135:0/3801578744 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda1c13b810 con 0x7fda1c0a9d70 2024-03-29T11:30:17.927 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.931+0000 7fda197fa640 1 -- 172.21.15.135:0/3801578744 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fda1001aa10 con 0x7fda1c0a9d70 2024-03-29T11:30:17.927 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.931+0000 7fda197fa640 1 -- 172.21.15.135:0/3801578744 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7fda10084ac0 con 0x7fda1c0a9d70 2024-03-29T11:30:17.927 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.931+0000 7fda197fa640 10 client.4451.objecter ms_dispatch 0x7fda1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:17.927 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.931+0000 7fda197fa640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:17.927 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.932+0000 7fda197fa640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:17.927 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.932+0000 7fda197fa640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-29T11:30:17.928 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.934+0000 7fda1bfff640 10 monclient: get_auth_request con 0x7fda0806dc80 auth_method 0 2024-03-29T11:30:17.928 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.935+0000 7fda197fa640 10 client.4451.objecter ms_handle_connect 0x7fda0806dc80 2024-03-29T11:30:17.931 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.938+0000 7fda197fa640 1 -- 172.21.15.135:0/3801578744 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fda100501d0 con 0x7fda1c0a9d70 2024-03-29T11:30:17.931 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.939+0000 7fda197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:17.931 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:17.939+0000 7fda197fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:18.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:18.259+0000 7fda22c33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:18.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:18.259+0000 7fda22c33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:18.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:18.259+0000 7fda22c33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:30:18.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:18.259+0000 7fda22c33640 1 -- 172.21.15.135:0/3801578744 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda1c075f10 con 0x7fda1c0a9d70 2024-03-29T11:30:18.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:18.261+0000 7fda197fa640 1 -- 172.21.15.135:0/3801578744 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7fda100522d0 con 0x7fda1c0a9d70 2024-03-29T11:30:18.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:18.261+0000 7fda197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:18.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:18.261+0000 7fda197fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:18.255 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:30:18.255 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:18.257 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:18.265+0000 7fda22c33640 10 librados: watch_flush enter 2024-03-29T11:30:18.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:18.266+0000 7fda22c33640 10 librados: watch_flush exit 2024-03-29T11:30:18.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:18.266+0000 7fda22c33640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-29T11:30:18.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:18.266+0000 7fda22c33640 10 client.4451.objecter successfully canceled tick 2024-03-29T11:30:18.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:18.266+0000 7fda22c33640 1 -- 172.21.15.135:0/3801578744 >> v1:172.21.15.135:6816/3469875285 conn(0x7fda0806dc80 legacy=0x7fda08070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:18.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:18.266+0000 7fda22c33640 10 monclient: shutdown 2024-03-29T11:30:18.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:18.266+0000 7fda22c33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:18.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:18.266+0000 7fda22c33640 1 -- 172.21.15.135:0/3801578744 >> v1:172.21.15.135:6790/0 conn(0x7fda1c0a9d70 legacy=0x7fda1c139690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:18.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:18.266+0000 7fda22c33640 1 -- 172.21.15.135:0/3801578744 shutdown_connections 2024-03-29T11:30:18.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:18.266+0000 7fda22c33640 1 -- 172.21.15.135:0/3801578744 >> 172.21.15.135:0/3801578744 conn(0x7fda1c066a00 msgr2=0x7fda1c0b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:18.260 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:18.266+0000 7fda22c33640 1 -- 172.21.15.135:0/3801578744 shutdown_connections 2024-03-29T11:30:18.260 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:18.267+0000 7fda22c33640 1 -- 172.21.15.135:0/3801578744 wait complete. 2024-03-29T11:30:18.260 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:18.267+0000 7fda22c33640 1 librados: shutdown 2024-03-29T11:30:18.277 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:25.084 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:25.246 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.252+0000 7fbb91263640 10 monclient: get_monmap_and_config 2024-03-29T11:30:25.246 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.252+0000 7fbb91263640 10 monclient: build_initial_monmap 2024-03-29T11:30:25.246 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.252+0000 7fbb91263640 10 monclient: monmap: 2024-03-29T11:30:25.246 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:30:25.246 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:25.246 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:30:25.254053+0000 2024-03-29T11:30:25.247 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:30:25.254053+0000 2024-03-29T11:30:25.247 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:25.247 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:25.247 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:30:25.247 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:30:25.247 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:30:25.247 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:25.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.253+0000 7fbb91263640 1 Processor -- start 2024-03-29T11:30:25.250 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.256+0000 7fbb91263640 1 -- start start 2024-03-29T11:30:25.250 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.256+0000 7fbb91263640 10 monclient: init 2024-03-29T11:30:25.250 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.257+0000 7fbb91263640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:25.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.258+0000 7fbb91263640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:25.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.258+0000 7fbb91263640 10 monclient(hunting): picked mon.noname-a con 0x7fbb8c0b0770 addr v1:172.21.15.135:6789/0 2024-03-29T11:30:25.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.261+0000 7fbb91263640 10 monclient(hunting): picked mon.noname-b con 0x7fbb8c14a700 addr v1:172.21.15.135:6790/0 2024-03-29T11:30:25.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.261+0000 7fbb8affd640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fbb8c0b0770 0x7fbb8c149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:40503/0 (socket says 172.21.15.135:40503) 2024-03-29T11:30:25.262 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.265+0000 7fbb8affd640 1 -- 172.21.15.135:0/2476205604 learned_addr learned my addr 172.21.15.135:0/2476205604 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:30:25.262 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.264+0000 7fbb91263640 10 monclient(hunting): picked mon.noname-c con 0x7fbb8c0ac5a0 addr v1:172.21.15.194:6789/0 2024-03-29T11:30:25.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.265+0000 7fbb91263640 1 -- 172.21.15.135:0/2476205604 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb8c0aba70 con 0x7fbb8c0b0770 2024-03-29T11:30:25.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.265+0000 7fbb91263640 1 -- 172.21.15.135:0/2476205604 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb8c0aecb0 con 0x7fbb8c0ac5a0 2024-03-29T11:30:25.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.265+0000 7fbb91263640 1 -- 172.21.15.135:0/2476205604 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb8c0af150 con 0x7fbb8c14a700 2024-03-29T11:30:25.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.265+0000 7fbb91263640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:25.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.265+0000 7fbb91263640 10 monclient(hunting): authenticate will time out at 1132.634651s 2024-03-29T11:30:25.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.265+0000 7fbb89ffb640 1 -- 172.21.15.135:0/2476205604 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811341979 0 0) 0x7fbb8c0aecb0 con 0x7fbb8c0ac5a0 2024-03-29T11:30:25.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.265+0000 7fbb89ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:25.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.265+0000 7fbb89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:25.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.265+0000 7fbb89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:25.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.265+0000 7fbb89ffb640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:30:25.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.265+0000 7fbb89ffb640 1 -- 172.21.15.135:0/2476205604 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb74003520 con 0x7fbb8c0ac5a0 2024-03-29T11:30:25.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.265+0000 7fbb89ffb640 1 -- 172.21.15.135:0/2476205604 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1084271762 0 0) 0x7fbb8c0aba70 con 0x7fbb8c0b0770 2024-03-29T11:30:25.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.265+0000 7fbb89ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:25.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.265+0000 7fbb89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:25.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.265+0000 7fbb89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:25.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.265+0000 7fbb89ffb640 10 monclient(hunting): my global_id is 4437 2024-03-29T11:30:25.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.265+0000 7fbb89ffb640 1 -- 172.21.15.135:0/2476205604 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb8c0aecb0 con 0x7fbb8c0b0770 2024-03-29T11:30:25.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.266+0000 7fbb89ffb640 1 -- 172.21.15.135:0/2476205604 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2246720203 0 0) 0x7fbb8c0aecb0 con 0x7fbb8c0b0770 2024-03-29T11:30:25.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.266+0000 7fbb89ffb640 1 -- 172.21.15.135:0/2476205604 >> v1:172.21.15.135:6790/0 conn(0x7fbb8c14a700 legacy=0x7fbb8c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:30:25.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.266+0000 7fbb89ffb640 1 -- 172.21.15.135:0/2476205604 >> v1:172.21.15.194:6789/0 conn(0x7fbb8c0ac5a0 legacy=0x7fbb8c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:25.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.266+0000 7fbb89ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:30:25.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.266+0000 7fbb89ffb640 1 monclient: found mon.noname-a 2024-03-29T11:30:25.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.266+0000 7fbb89ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:25.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.266+0000 7fbb89ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.135:6789/0 2024-03-29T11:30:25.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.266+0000 7fbb89ffb640 1 -- 172.21.15.135:0/2476205604 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb8c0ac9d0 con 0x7fbb8c0b0770 2024-03-29T11:30:25.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.266+0000 7fbb89ffb640 10 monclient: _finish_auth 0 2024-03-29T11:30:25.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.266+0000 7fbb89ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:30:25.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.266+0000 7fbb89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:25.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.266+0000 7fbb89ffb640 1 -- 172.21.15.135:0/2476205604 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fbb800015f0 con 0x7fbb8c0b0770 2024-03-29T11:30:25.265 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.266+0000 7fbb89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:25.265 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.266+0000 7fbb89ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:25.265 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.266+0000 7fbb89ffb640 10 monclient: dump: 2024-03-29T11:30:25.265 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:25.265 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:25.265 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:25.265 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:25.265 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:25.265 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:25.265 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:25.265 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:25.265 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:25.265 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:25.265 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.266+0000 7fbb89ffb640 1 -- 172.21.15.135:0/2476205604 <== mon.0 v1:172.21.15.135:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb80001750 con 0x7fbb8c0b0770 2024-03-29T11:30:25.265 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.266+0000 7fbb89ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:25.265 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.266+0000 7fbb89ffb640 1 -- 172.21.15.135:0/2476205604 <== mon.0 v1:172.21.15.135:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fbb80001b60 con 0x7fbb8c0b0770 2024-03-29T11:30:25.266 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.266+0000 7fbb89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:25.266 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.266+0000 7fbb89ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:25.266 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.266+0000 7fbb89ffb640 10 monclient: dump: 2024-03-29T11:30:25.266 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:25.266 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:25.266 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:25.266 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:25.266 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:25.266 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:25.266 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:25.266 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:25.266 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:25.266 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:25.266 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.266+0000 7fbb91263640 5 monclient: authenticate success, global_id 4437 2024-03-29T11:30:25.267 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.266+0000 7fbb91263640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:25.267 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.266+0000 7fbb91263640 4 set_mon_vals no callback set 2024-03-29T11:30:25.267 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.267+0000 7fbb91263640 10 monclient: shutdown 2024-03-29T11:30:25.267 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.268+0000 7fbb91263640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:25.267 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.268+0000 7fbb91263640 1 -- 172.21.15.135:0/2476205604 >> v1:172.21.15.135:6789/0 conn(0x7fbb8c0b0770 legacy=0x7fbb8c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:25.267 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.268+0000 7fbb91263640 1 -- 172.21.15.135:0/2476205604 shutdown_connections 2024-03-29T11:30:25.267 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.268+0000 7fbb91263640 1 -- 172.21.15.135:0/2476205604 >> 172.21.15.135:0/2476205604 conn(0x7fbb8c06b660 msgr2=0x7fbb8c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:25.267 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.268+0000 7fbb91263640 1 -- 172.21.15.135:0/2476205604 shutdown_connections 2024-03-29T11:30:25.267 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.268+0000 7fbb91263640 1 -- 172.21.15.135:0/2476205604 wait complete. 2024-03-29T11:30:25.267 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.269+0000 7fbb91263640 10 monclient: build_initial_monmap 2024-03-29T11:30:25.267 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.269+0000 7fbb91263640 10 monclient: monmap: 2024-03-29T11:30:25.267 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:30:25.267 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:25.267 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:30:25.268 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:30:25.268 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:25.268 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:25.268 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:30:25.268 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:30:25.268 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:30:25.268 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:25.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.269+0000 7fbb91263640 1 librados: starting msgr at 2024-03-29T11:30:25.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.269+0000 7fbb91263640 1 librados: starting objecter 2024-03-29T11:30:25.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.270+0000 7fbb91263640 1 Processor -- start 2024-03-29T11:30:25.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.272+0000 7fbb91263640 1 -- start start 2024-03-29T11:30:25.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.272+0000 7fbb91263640 1 librados: setting wanted keys 2024-03-29T11:30:25.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.272+0000 7fbb91263640 1 librados: calling monclient init 2024-03-29T11:30:25.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.272+0000 7fbb91263640 10 monclient: init 2024-03-29T11:30:25.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.273+0000 7fbb91263640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:25.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.273+0000 7fbb91263640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:30:25.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.275+0000 7fbb91263640 10 monclient(hunting): picked mon.noname-b con 0x7fbb8c0ac5a0 addr v1:172.21.15.194:6789/0 2024-03-29T11:30:25.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.276+0000 7fbb8affd640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fbb8c0ac5a0 0x7fbb8c1265d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:46727/0 (socket says 172.21.15.135:46727) 2024-03-29T11:30:25.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.276+0000 7fbb8affd640 1 -- 172.21.15.135:0/4219270417 learned_addr learned my addr 172.21.15.135:0/4219270417 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:30:25.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.276+0000 7fbb91263640 10 monclient(hunting): picked mon.noname-c con 0x7fbb8c0b0770 addr v1:172.21.15.135:6790/0 2024-03-29T11:30:25.271 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.277+0000 7fbb91263640 10 monclient(hunting): picked mon.noname-a con 0x7fbb8c14a700 addr v1:172.21.15.135:6789/0 2024-03-29T11:30:25.271 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.277+0000 7fbb91263640 1 -- 172.21.15.135:0/4219270417 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb8c145970 con 0x7fbb8c14a700 2024-03-29T11:30:25.271 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.277+0000 7fbb91263640 1 -- 172.21.15.135:0/4219270417 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb8c127ee0 con 0x7fbb8c0ac5a0 2024-03-29T11:30:25.271 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.277+0000 7fbb91263640 1 -- 172.21.15.135:0/4219270417 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb8c125b60 con 0x7fbb8c0b0770 2024-03-29T11:30:25.271 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.277+0000 7fbb91263640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:25.271 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.277+0000 7fbb91263640 10 monclient(hunting): authenticate will time out at 1132.646181s 2024-03-29T11:30:25.272 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.277+0000 7fbb737fe640 10 client.?.objecter ms_handle_connect 0x7fbb8c0ac5a0 2024-03-29T11:30:25.272 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.277+0000 7fbb737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:25.272 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.277+0000 7fbb737fe640 1 -- 172.21.15.135:0/4219270417 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2580238207 0 0) 0x7fbb8c127ee0 con 0x7fbb8c0ac5a0 2024-03-29T11:30:25.272 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.277+0000 7fbb737fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:25.272 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.277+0000 7fbb737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:25.273 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.277+0000 7fbb737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:25.273 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.277+0000 7fbb737fe640 10 monclient(hunting): my global_id is 4444 2024-03-29T11:30:25.273 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.277+0000 7fbb737fe640 1 -- 172.21.15.135:0/4219270417 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb78004eb0 con 0x7fbb8c0ac5a0 2024-03-29T11:30:25.273 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.278+0000 7fbb737fe640 1 -- 172.21.15.135:0/4219270417 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3295455515 0 0) 0x7fbb78004eb0 con 0x7fbb8c0ac5a0 2024-03-29T11:30:25.273 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.278+0000 7fbb737fe640 1 -- 172.21.15.135:0/4219270417 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb8c127ee0 con 0x7fbb8c0ac5a0 2024-03-29T11:30:25.273 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.278+0000 7fbb737fe640 1 -- 172.21.15.135:0/4219270417 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fbb80001b60 con 0x7fbb8c0ac5a0 2024-03-29T11:30:25.273 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.278+0000 7fbb737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:25.273 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.278+0000 7fbb737fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:25.273 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.279+0000 7fbb737fe640 10 monclient(hunting): dump: 2024-03-29T11:30:25.273 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:25.273 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:25.273 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:25.273 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:25.273 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:25.273 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:25.274 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:25.274 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:25.274 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:25.274 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:25.274 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.279+0000 7fbb737fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:25.274 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.279+0000 7fbb737fe640 1 -- 172.21.15.135:0/4219270417 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1465784039 0 0) 0x7fbb8c127ee0 con 0x7fbb8c0ac5a0 2024-03-29T11:30:25.274 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.279+0000 7fbb737fe640 1 -- 172.21.15.135:0/4219270417 >> v1:172.21.15.135:6790/0 conn(0x7fbb8c0b0770 legacy=0x7fbb8c1ded40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:25.274 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.279+0000 7fbb737fe640 1 -- 172.21.15.135:0/4219270417 >> v1:172.21.15.135:6789/0 conn(0x7fbb8c14a700 legacy=0x7fbb8c1e5fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:25.277 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.280+0000 7fbb737fe640 10 monclient: _finish_hunting 0 2024-03-29T11:30:25.277 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.280+0000 7fbb737fe640 1 monclient: found mon.b 2024-03-29T11:30:25.278 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.280+0000 7fbb737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:25.278 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.280+0000 7fbb737fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:30:25.278 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.280+0000 7fbb737fe640 1 -- 172.21.15.135:0/4219270417 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb8c126ce0 con 0x7fbb8c0ac5a0 2024-03-29T11:30:25.278 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.280+0000 7fbb737fe640 10 monclient: _finish_auth 0 2024-03-29T11:30:25.278 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.280+0000 7fbb737fe640 10 monclient: _check_auth_tickets 2024-03-29T11:30:25.278 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.280+0000 7fbb737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:25.278 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.280+0000 7fbb737fe640 1 -- 172.21.15.135:0/4219270417 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb80004b00 con 0x7fbb8c0ac5a0 2024-03-29T11:30:25.278 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.280+0000 7fbb91263640 5 monclient: authenticate success, global_id 4444 2024-03-29T11:30:25.278 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.280+0000 7fbb91263640 10 monclient: _renew_subs 2024-03-29T11:30:25.278 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.280+0000 7fbb91263640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:30:25.278 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.280+0000 7fbb91263640 1 -- 172.21.15.135:0/4219270417 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb8c1df450 con 0x7fbb8c0ac5a0 2024-03-29T11:30:25.278 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.280+0000 7fbb91263640 10 client.4444.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:25.278 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.280+0000 7fbb91263640 10 monclient: _renew_subs 2024-03-29T11:30:25.279 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.281+0000 7fbb91263640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:30:25.279 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.281+0000 7fbb91263640 1 -- 172.21.15.135:0/4219270417 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb8c1e76e0 con 0x7fbb8c0ac5a0 2024-03-29T11:30:25.279 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.281+0000 7fbb91263640 1 librados: init done 2024-03-29T11:30:25.279 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.281+0000 7fbb737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:25.279 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.281+0000 7fbb737fe640 1 -- 172.21.15.135:0/4219270417 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fbb80004f50 con 0x7fbb8c0ac5a0 2024-03-29T11:30:25.279 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.281+0000 7fbb737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:25.279 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.281+0000 7fbb737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:25.279 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.281+0000 7fbb737fe640 10 monclient: dump: 2024-03-29T11:30:25.279 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:25.279 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:25.279 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:25.279 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:25.279 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:25.279 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:25.279 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:25.280 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:25.280 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:25.280 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:25.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.281+0000 7fbb897fa640 4 set_mon_vals no callback set 2024-03-29T11:30:25.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.282+0000 7fbb91263640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:25.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.282+0000 7fbb91263640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:25.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.282+0000 7fbb91263640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:30:25.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.282+0000 7fbb91263640 1 -- 172.21.15.135:0/4219270417 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb8c0ab560 con 0x7fbb8c0ac5a0 2024-03-29T11:30:25.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.283+0000 7fbb737fe640 1 -- 172.21.15.135:0/4219270417 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fbb800050b0 con 0x7fbb8c0ac5a0 2024-03-29T11:30:25.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.283+0000 7fbb737fe640 1 -- 172.21.15.135:0/4219270417 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7fbb80084d20 con 0x7fbb8c0ac5a0 2024-03-29T11:30:25.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.283+0000 7fbb8a7fc640 10 monclient: get_auth_request con 0x7fbb7806dc90 auth_method 0 2024-03-29T11:30:25.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.283+0000 7fbb737fe640 10 client.4444.objecter ms_dispatch 0x7fbb8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:25.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.283+0000 7fbb737fe640 3 client.4444.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:25.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.283+0000 7fbb737fe640 3 client.4444.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:25.280 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.283+0000 7fbb737fe640 20 client.4444.objecter dump_active .. 0 homeless 2024-03-29T11:30:25.281 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.284+0000 7fbb737fe640 10 client.4444.objecter ms_handle_connect 0x7fbb7806dc90 2024-03-29T11:30:25.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.289+0000 7fbb737fe640 1 -- 172.21.15.135:0/4219270417 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbb80050600 con 0x7fbb8c0ac5a0 2024-03-29T11:30:25.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.289+0000 7fbb737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:25.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.289+0000 7fbb737fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:25.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.607+0000 7fbb91263640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:25.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.607+0000 7fbb91263640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:25.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.607+0000 7fbb91263640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:30:25.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.607+0000 7fbb91263640 1 -- 172.21.15.135:0/4219270417 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb8c075f10 con 0x7fbb8c0ac5a0 2024-03-29T11:30:25.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.608+0000 7fbb737fe640 1 -- 172.21.15.135:0/4219270417 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7fbb80052530 con 0x7fbb8c0ac5a0 2024-03-29T11:30:25.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.608+0000 7fbb737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:25.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.608+0000 7fbb737fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:25.603 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:30:25.603 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:25.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.612+0000 7fbb91263640 10 librados: watch_flush enter 2024-03-29T11:30:25.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.612+0000 7fbb91263640 10 librados: watch_flush exit 2024-03-29T11:30:25.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.613+0000 7fbb91263640 20 client.4444.objecter shutdown clearing up homeless session... 2024-03-29T11:30:25.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.613+0000 7fbb91263640 10 client.4444.objecter successfully canceled tick 2024-03-29T11:30:25.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.613+0000 7fbb91263640 1 -- 172.21.15.135:0/4219270417 >> v1:172.21.15.135:6816/3469875285 conn(0x7fbb7806dc90 legacy=0x7fbb78070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:25.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.613+0000 7fbb91263640 10 monclient: shutdown 2024-03-29T11:30:25.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.613+0000 7fbb91263640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:25.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.613+0000 7fbb91263640 1 -- 172.21.15.135:0/4219270417 >> v1:172.21.15.194:6789/0 conn(0x7fbb8c0ac5a0 legacy=0x7fbb8c1265d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:25.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.613+0000 7fbb91263640 1 -- 172.21.15.135:0/4219270417 shutdown_connections 2024-03-29T11:30:25.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.614+0000 7fbb91263640 1 -- 172.21.15.135:0/4219270417 >> 172.21.15.135:0/4219270417 conn(0x7fbb8c06b660 msgr2=0x7fbb8c06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:25.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.615+0000 7fbb91263640 1 -- 172.21.15.135:0/4219270417 shutdown_connections 2024-03-29T11:30:25.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.616+0000 7fbb91263640 1 -- 172.21.15.135:0/4219270417 wait complete. 2024-03-29T11:30:25.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:25.616+0000 7fbb91263640 1 librados: shutdown 2024-03-29T11:30:25.626 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:32.433 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:32.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.591+0000 7f8f59aae640 10 monclient: get_monmap_and_config 2024-03-29T11:30:32.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.591+0000 7f8f59aae640 10 monclient: build_initial_monmap 2024-03-29T11:30:32.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.591+0000 7f8f59aae640 10 monclient: monmap: 2024-03-29T11:30:32.587 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:30:32.587 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:32.587 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:30:32.593179+0000 2024-03-29T11:30:32.587 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:30:32.593179+0000 2024-03-29T11:30:32.587 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:32.587 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:32.587 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:30:32.587 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:30:32.587 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:30:32.587 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:32.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.592+0000 7f8f59aae640 1 Processor -- start 2024-03-29T11:30:32.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.592+0000 7f8f59aae640 1 -- start start 2024-03-29T11:30:32.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.592+0000 7f8f59aae640 10 monclient: init 2024-03-29T11:30:32.597 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.593+0000 7f8f59aae640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:32.597 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.593+0000 7f8f59aae640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:30:32.598 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.593+0000 7f8f59aae640 10 monclient(hunting): picked mon.noname-c con 0x7f8f540af6c0 addr v1:172.21.15.194:6789/0 2024-03-29T11:30:32.598 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.593+0000 7f8f59aae640 10 monclient(hunting): picked mon.noname-b con 0x7f8f5414f6b0 addr v1:172.21.15.135:6790/0 2024-03-29T11:30:32.598 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.593+0000 7f8f59aae640 10 monclient(hunting): picked mon.noname-a con 0x7f8f54152980 addr v1:172.21.15.135:6789/0 2024-03-29T11:30:32.598 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.593+0000 7f8f59aae640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f540a94e0 con 0x7f8f54152980 2024-03-29T11:30:32.598 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.593+0000 7f8f59aae640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f540afb60 con 0x7f8f540af6c0 2024-03-29T11:30:32.598 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.593+0000 7f8f59aae640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f540aab80 con 0x7f8f5414f6b0 2024-03-29T11:30:32.598 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.593+0000 7f8f59aae640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:32.598 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.593+0000 7f8f59aae640 10 monclient(hunting): authenticate will time out at 1139.962781s 2024-03-29T11:30:32.598 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.599+0000 7f8f52ffd640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f8f5414f6b0 0x7f8f5414fae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:56021/0 (socket says 172.21.15.135:56021) 2024-03-29T11:30:32.598 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.599+0000 7f8f52ffd640 1 -- 172.21.15.135:0/2126575150 learned_addr learned my addr 172.21.15.135:0/2126575150 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:30:32.598 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.600+0000 7f8f527fc640 1 -- 172.21.15.135:0/2126575150 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1458437143 0 0) 0x7f8f540afb60 con 0x7f8f540af6c0 2024-03-29T11:30:32.598 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.600+0000 7f8f527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:32.598 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.600+0000 7f8f527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:32.598 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.600+0000 7f8f527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:32.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.600+0000 7f8f527fc640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:30:32.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.600+0000 7f8f527fc640 1 -- 172.21.15.135:0/2126575150 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f3c003520 con 0x7f8f540af6c0 2024-03-29T11:30:32.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.600+0000 7f8f527fc640 1 -- 172.21.15.135:0/2126575150 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3960639550 0 0) 0x7f8f3c003520 con 0x7f8f540af6c0 2024-03-29T11:30:32.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.600+0000 7f8f527fc640 1 -- 172.21.15.135:0/2126575150 >> v1:172.21.15.135:6790/0 conn(0x7f8f5414f6b0 legacy=0x7f8f5414fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:30:32.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.600+0000 7f8f527fc640 1 -- 172.21.15.135:0/2126575150 >> v1:172.21.15.135:6789/0 conn(0x7f8f54152980 legacy=0x7f8f54152e10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:32.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.601+0000 7f8f527fc640 10 monclient: _finish_hunting 0 2024-03-29T11:30:32.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.601+0000 7f8f527fc640 1 monclient: found mon.noname-c 2024-03-29T11:30:32.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.601+0000 7f8f527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:32.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.601+0000 7f8f527fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:30:32.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.601+0000 7f8f527fc640 1 -- 172.21.15.135:0/2126575150 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f540aa680 con 0x7f8f540af6c0 2024-03-29T11:30:32.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.601+0000 7f8f527fc640 10 monclient: _finish_auth 0 2024-03-29T11:30:32.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.601+0000 7f8f527fc640 10 monclient: _check_auth_tickets 2024-03-29T11:30:32.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.601+0000 7f8f527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:32.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.601+0000 7f8f527fc640 1 -- 172.21.15.135:0/2126575150 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f8f44001670 con 0x7f8f540af6c0 2024-03-29T11:30:32.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.601+0000 7f8f527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:32.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.601+0000 7f8f527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:32.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.601+0000 7f8f527fc640 10 monclient: dump: 2024-03-29T11:30:32.601 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:32.601 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:32.601 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:32.602 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:32.602 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:32.602 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:32.602 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:32.602 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:32.602 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:32.602 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:32.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.601+0000 7f8f59aae640 5 monclient: authenticate success, global_id 4447 2024-03-29T11:30:32.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.601+0000 7f8f59aae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:32.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.601+0000 7f8f527fc640 1 -- 172.21.15.135:0/2126575150 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f440018d0 con 0x7f8f540af6c0 2024-03-29T11:30:32.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.601+0000 7f8f527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:32.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.601+0000 7f8f59aae640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:32.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.601+0000 7f8f59aae640 4 set_mon_vals no callback set 2024-03-29T11:30:32.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.602+0000 7f8f59aae640 10 monclient: shutdown 2024-03-29T11:30:32.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.602+0000 7f8f59aae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:32.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.602+0000 7f8f59aae640 1 -- 172.21.15.135:0/2126575150 >> v1:172.21.15.194:6789/0 conn(0x7f8f540af6c0 legacy=0x7f8f540ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:32.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.602+0000 7f8f527fc640 1 -- 172.21.15.135:0/2126575150 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f8f44001d20 con 0x7f8f540af6c0 2024-03-29T11:30:32.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.602+0000 7f8f527fc640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:30:32.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.602+0000 7f8f59aae640 1 -- 172.21.15.135:0/2126575150 shutdown_connections 2024-03-29T11:30:32.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.602+0000 7f8f59aae640 1 -- 172.21.15.135:0/2126575150 >> 172.21.15.135:0/2126575150 conn(0x7f8f540b35e0 msgr2=0x7f8f540b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:32.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.604+0000 7f8f59aae640 1 -- 172.21.15.135:0/2126575150 shutdown_connections 2024-03-29T11:30:32.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.605+0000 7f8f59aae640 1 -- 172.21.15.135:0/2126575150 wait complete. 2024-03-29T11:30:32.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.606+0000 7f8f59aae640 10 monclient: build_initial_monmap 2024-03-29T11:30:32.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.606+0000 7f8f59aae640 10 monclient: monmap: 2024-03-29T11:30:32.603 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:30:32.603 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:32.603 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:30:32.603 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:30:32.603 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:32.603 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:32.604 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:30:32.604 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:30:32.604 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:30:32.604 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:32.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.607+0000 7f8f59aae640 1 librados: starting msgr at 2024-03-29T11:30:32.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.607+0000 7f8f59aae640 1 librados: starting objecter 2024-03-29T11:30:32.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.607+0000 7f8f59aae640 1 Processor -- start 2024-03-29T11:30:32.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.607+0000 7f8f59aae640 1 -- start start 2024-03-29T11:30:32.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.607+0000 7f8f59aae640 1 librados: setting wanted keys 2024-03-29T11:30:32.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.607+0000 7f8f59aae640 1 librados: calling monclient init 2024-03-29T11:30:32.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.607+0000 7f8f59aae640 10 monclient: init 2024-03-29T11:30:32.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.608+0000 7f8f59aae640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:32.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.608+0000 7f8f59aae640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:30:32.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.608+0000 7f8f59aae640 10 monclient(hunting): picked mon.noname-c con 0x7f8f540af6c0 addr v1:172.21.15.135:6790/0 2024-03-29T11:30:32.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.608+0000 7f8f59aae640 10 monclient(hunting): picked mon.noname-b con 0x7f8f5414f6b0 addr v1:172.21.15.194:6789/0 2024-03-29T11:30:32.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.608+0000 7f8f59aae640 10 monclient(hunting): picked mon.noname-a con 0x7f8f54152980 addr v1:172.21.15.135:6789/0 2024-03-29T11:30:32.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.608+0000 7f8f59aae640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f541482a0 con 0x7f8f54152980 2024-03-29T11:30:32.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.608+0000 7f8f59aae640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f54148580 con 0x7f8f5414f6b0 2024-03-29T11:30:32.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.608+0000 7f8f59aae640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f54148860 con 0x7f8f540af6c0 2024-03-29T11:30:32.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.608+0000 7f8f59aae640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:32.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.608+0000 7f8f59aae640 10 monclient(hunting): authenticate will time out at 1139.977680s 2024-03-29T11:30:32.611 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.614+0000 7f8f53fff640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f8f54152980 0x7f8f541f2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:40283/0 (socket says 172.21.15.135:40283) 2024-03-29T11:30:32.612 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.614+0000 7f8f53fff640 1 -- 172.21.15.135:0/3933433907 learned_addr learned my addr 172.21.15.135:0/3933433907 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:30:32.612 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.614+0000 7f8f3bfff640 10 client.?.objecter ms_handle_connect 0x7f8f54152980 2024-03-29T11:30:32.612 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.614+0000 7f8f3bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:32.612 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.614+0000 7f8f3bfff640 10 client.?.objecter ms_handle_connect 0x7f8f540af6c0 2024-03-29T11:30:32.612 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.614+0000 7f8f3bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:32.612 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.614+0000 7f8f3bfff640 1 -- 172.21.15.135:0/3933433907 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097008838 0 0) 0x7f8f541482a0 con 0x7f8f54152980 2024-03-29T11:30:32.612 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.614+0000 7f8f3bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:32.612 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.614+0000 7f8f3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:32.612 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.614+0000 7f8f3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:32.612 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.614+0000 7f8f3bfff640 10 monclient(hunting): my global_id is 4440 2024-03-29T11:30:32.612 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.614+0000 7f8f3bfff640 1 -- 172.21.15.135:0/3933433907 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f40004eb0 con 0x7f8f54152980 2024-03-29T11:30:32.612 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.616+0000 7f8f3bfff640 10 client.?.objecter ms_handle_connect 0x7f8f5414f6b0 2024-03-29T11:30:32.612 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.616+0000 7f8f3bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:32.612 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.617+0000 7f8f3bfff640 1 -- 172.21.15.135:0/3933433907 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2335455606 0 0) 0x7f8f40004eb0 con 0x7f8f54152980 2024-03-29T11:30:32.613 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.617+0000 7f8f3bfff640 1 -- 172.21.15.135:0/3933433907 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f40005300 con 0x7f8f54152980 2024-03-29T11:30:32.613 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.617+0000 7f8f3bfff640 1 -- 172.21.15.135:0/3933433907 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f8f48001710 con 0x7f8f54152980 2024-03-29T11:30:32.613 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.617+0000 7f8f3bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:32.613 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.617+0000 7f8f3bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:32.613 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.617+0000 7f8f3bfff640 10 monclient(hunting): dump: 2024-03-29T11:30:32.613 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:32.613 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:32.613 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:32.613 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:32.613 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:32.613 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:32.613 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:32.613 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:32.613 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:32.614 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:32.614 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.617+0000 7f8f3bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:32.614 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.617+0000 7f8f3bfff640 1 -- 172.21.15.135:0/3933433907 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2890072170 0 0) 0x7f8f54148580 con 0x7f8f5414f6b0 2024-03-29T11:30:32.614 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.617+0000 7f8f3bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:32.614 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.617+0000 7f8f3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:32.614 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.617+0000 7f8f3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:32.614 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.617+0000 7f8f3bfff640 10 monclient(hunting): my global_id is 4450 2024-03-29T11:30:32.614 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.617+0000 7f8f3bfff640 1 -- 172.21.15.135:0/3933433907 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f40005850 con 0x7f8f5414f6b0 2024-03-29T11:30:32.615 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.618+0000 7f8f3bfff640 1 -- 172.21.15.135:0/3933433907 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 816575191 0 0) 0x7f8f40005850 con 0x7f8f5414f6b0 2024-03-29T11:30:32.615 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.618+0000 7f8f3bfff640 1 -- 172.21.15.135:0/3933433907 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f40005ce0 con 0x7f8f5414f6b0 2024-03-29T11:30:32.616 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.618+0000 7f8f3bfff640 1 -- 172.21.15.135:0/3933433907 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f8f4c001810 con 0x7f8f5414f6b0 2024-03-29T11:30:32.616 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.618+0000 7f8f3bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:32.616 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.618+0000 7f8f3bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:32.616 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.618+0000 7f8f3bfff640 10 monclient(hunting): dump: 2024-03-29T11:30:32.616 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:32.616 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:32.616 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:32.616 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:32.616 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:32.616 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:32.616 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:32.616 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:32.616 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:32.617 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:32.617 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.618+0000 7f8f3bfff640 1 -- 172.21.15.135:0/3933433907 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693463115 0 0) 0x7f8f54148860 con 0x7f8f540af6c0 2024-03-29T11:30:32.617 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.618+0000 7f8f3bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:32.617 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.618+0000 7f8f3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:32.617 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.618+0000 7f8f3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:32.617 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.618+0000 7f8f3bfff640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:30:32.617 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.618+0000 7f8f3bfff640 1 -- 172.21.15.135:0/3933433907 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f40005850 con 0x7f8f540af6c0 2024-03-29T11:30:32.617 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.618+0000 7f8f3bfff640 1 -- 172.21.15.135:0/3933433907 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2592659880 0 0) 0x7f8f40005ce0 con 0x7f8f5414f6b0 2024-03-29T11:30:32.617 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.618+0000 7f8f3bfff640 1 -- 172.21.15.135:0/3933433907 >> v1:172.21.15.135:6790/0 conn(0x7f8f540af6c0 legacy=0x7f8f54146800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:32.617 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.618+0000 7f8f3bfff640 1 -- 172.21.15.135:0/3933433907 >> v1:172.21.15.135:6789/0 conn(0x7f8f54152980 legacy=0x7f8f541f2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:32.617 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.618+0000 7f8f3bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:30:32.617 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.618+0000 7f8f3bfff640 1 monclient: found mon.b 2024-03-29T11:30:32.617 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.618+0000 7f8f3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:32.617 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.618+0000 7f8f3bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:30:32.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.618+0000 7f8f3bfff640 1 -- 172.21.15.135:0/3933433907 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f54148b40 con 0x7f8f5414f6b0 2024-03-29T11:30:32.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.618+0000 7f8f3bfff640 10 monclient: _finish_auth 0 2024-03-29T11:30:32.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.618+0000 7f8f3bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:30:32.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.618+0000 7f8f3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:32.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.619+0000 7f8f3bfff640 1 -- 172.21.15.135:0/3933433907 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f4c001d40 con 0x7f8f5414f6b0 2024-03-29T11:30:32.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.619+0000 7f8f3bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:32.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.619+0000 7f8f3bfff640 1 -- 172.21.15.135:0/3933433907 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f8f4c004dd0 con 0x7f8f5414f6b0 2024-03-29T11:30:32.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.619+0000 7f8f3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:32.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.619+0000 7f8f3bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:32.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.619+0000 7f8f3bfff640 10 monclient: dump: 2024-03-29T11:30:32.618 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:32.618 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:32.618 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:32.618 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:32.618 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:32.619 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:32.619 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:32.619 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:32.619 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:32.619 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:32.619 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.619+0000 7f8f59aae640 5 monclient: authenticate success, global_id 4450 2024-03-29T11:30:32.619 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.619+0000 7f8f59aae640 10 monclient: _renew_subs 2024-03-29T11:30:32.619 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.619+0000 7f8f59aae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:30:32.619 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.619+0000 7f8f59aae640 1 -- 172.21.15.135:0/3933433907 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f54147090 con 0x7f8f5414f6b0 2024-03-29T11:30:32.619 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.619+0000 7f8f517fa640 4 set_mon_vals no callback set 2024-03-29T11:30:32.619 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.619+0000 7f8f59aae640 10 client.4450.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:32.619 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.619+0000 7f8f59aae640 10 monclient: _renew_subs 2024-03-29T11:30:32.619 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.619+0000 7f8f59aae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:30:32.619 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.619+0000 7f8f59aae640 1 -- 172.21.15.135:0/3933433907 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f54088a90 con 0x7f8f5414f6b0 2024-03-29T11:30:32.619 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.619+0000 7f8f59aae640 1 librados: init done 2024-03-29T11:30:32.621 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.621+0000 7f8f59aae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:32.621 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.621+0000 7f8f59aae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:32.621 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.621+0000 7f8f59aae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:30:32.621 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.621+0000 7f8f59aae640 1 -- 172.21.15.135:0/3933433907 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f54088d40 con 0x7f8f5414f6b0 2024-03-29T11:30:32.621 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.622+0000 7f8f3bfff640 1 -- 172.21.15.135:0/3933433907 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f8f4c01ada0 con 0x7f8f5414f6b0 2024-03-29T11:30:32.621 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.622+0000 7f8f537fe640 10 monclient: get_auth_request con 0x7f8f4006e9e0 auth_method 0 2024-03-29T11:30:32.621 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.622+0000 7f8f3bfff640 1 -- 172.21.15.135:0/3933433907 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f8f4c001fb0 con 0x7f8f5414f6b0 2024-03-29T11:30:32.621 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.622+0000 7f8f3bfff640 10 client.4450.objecter ms_dispatch 0x7f8f54000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:32.621 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.622+0000 7f8f3bfff640 3 client.4450.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:32.621 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.622+0000 7f8f3bfff640 3 client.4450.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:32.621 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.622+0000 7f8f3bfff640 20 client.4450.objecter dump_active .. 0 homeless 2024-03-29T11:30:32.622 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.623+0000 7f8f3bfff640 10 client.4450.objecter ms_handle_connect 0x7f8f4006e9e0 2024-03-29T11:30:32.624 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.630+0000 7f8f3bfff640 1 -- 172.21.15.135:0/3933433907 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8f4c050a30 con 0x7f8f5414f6b0 2024-03-29T11:30:32.624 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.630+0000 7f8f3bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:32.624 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.630+0000 7f8f3bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:32.946 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.953+0000 7f8f59aae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:32.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.953+0000 7f8f59aae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:32.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.953+0000 7f8f59aae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:30:32.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.953+0000 7f8f59aae640 1 -- 172.21.15.135:0/3933433907 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f54075f10 con 0x7f8f5414f6b0 2024-03-29T11:30:32.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.954+0000 7f8f3bfff640 1 -- 172.21.15.135:0/3933433907 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f8f4c050bd0 con 0x7f8f5414f6b0 2024-03-29T11:30:32.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.954+0000 7f8f3bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:32.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.954+0000 7f8f3bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:32.948 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:30:32.948 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:32.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.957+0000 7f8f59aae640 10 librados: watch_flush enter 2024-03-29T11:30:32.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.957+0000 7f8f59aae640 10 librados: watch_flush exit 2024-03-29T11:30:32.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.957+0000 7f8f59aae640 20 client.4450.objecter shutdown clearing up homeless session... 2024-03-29T11:30:32.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.957+0000 7f8f59aae640 10 client.4450.objecter successfully canceled tick 2024-03-29T11:30:32.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.958+0000 7f8f59aae640 1 -- 172.21.15.135:0/3933433907 >> v1:172.21.15.135:6816/3469875285 conn(0x7f8f4006e9e0 legacy=0x7f8f40070ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:32.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.958+0000 7f8f59aae640 10 monclient: shutdown 2024-03-29T11:30:32.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.958+0000 7f8f59aae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:32.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.958+0000 7f8f59aae640 1 -- 172.21.15.135:0/3933433907 >> v1:172.21.15.194:6789/0 conn(0x7f8f5414f6b0 legacy=0x7f8f54070ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:32.952 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.958+0000 7f8f59aae640 1 -- 172.21.15.135:0/3933433907 shutdown_connections 2024-03-29T11:30:32.952 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.958+0000 7f8f59aae640 1 -- 172.21.15.135:0/3933433907 >> 172.21.15.135:0/3933433907 conn(0x7f8f540b35e0 msgr2=0x7f8f540ad240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:32.952 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.959+0000 7f8f59aae640 1 -- 172.21.15.135:0/3933433907 shutdown_connections 2024-03-29T11:30:32.952 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.959+0000 7f8f59aae640 1 -- 172.21.15.135:0/3933433907 wait complete. 2024-03-29T11:30:32.953 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:32.959+0000 7f8f59aae640 1 librados: shutdown 2024-03-29T11:30:32.973 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:39.780 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:39.931 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.937+0000 7f4f3a293640 10 monclient: get_monmap_and_config 2024-03-29T11:30:39.931 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.937+0000 7f4f3a293640 10 monclient: build_initial_monmap 2024-03-29T11:30:39.931 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.937+0000 7f4f3a293640 10 monclient: monmap: 2024-03-29T11:30:39.931 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:30:39.931 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:39.932 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:30:39.939059+0000 2024-03-29T11:30:39.932 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:30:39.939059+0000 2024-03-29T11:30:39.932 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:39.932 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:39.932 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:30:39.932 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:30:39.932 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:30:39.932 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:39.933 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.938+0000 7f4f3a293640 1 Processor -- start 2024-03-29T11:30:39.933 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.939+0000 7f4f3a293640 1 -- start start 2024-03-29T11:30:39.933 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.939+0000 7f4f3a293640 10 monclient: init 2024-03-29T11:30:39.933 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.939+0000 7f4f3a293640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:39.933 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.939+0000 7f4f3a293640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:30:39.933 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.939+0000 7f4f3a293640 10 monclient(hunting): picked mon.noname-b con 0x7f4f340b0770 addr v1:172.21.15.135:6790/0 2024-03-29T11:30:39.933 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.939+0000 7f4f3a293640 10 monclient(hunting): picked mon.noname-c con 0x7f4f3414a700 addr v1:172.21.15.194:6789/0 2024-03-29T11:30:39.933 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.940+0000 7f4f3a293640 10 monclient(hunting): picked mon.noname-a con 0x7f4f340ac5a0 addr v1:172.21.15.135:6789/0 2024-03-29T11:30:39.933 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.940+0000 7f4f3a293640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f340aba70 con 0x7f4f340ac5a0 2024-03-29T11:30:39.934 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.940+0000 7f4f3a293640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f340aecb0 con 0x7f4f3414a700 2024-03-29T11:30:39.934 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.940+0000 7f4f3a293640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f340af150 con 0x7f4f340b0770 2024-03-29T11:30:39.934 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.940+0000 7f4f3a293640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:39.934 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.940+0000 7f4f3a293640 10 monclient(hunting): authenticate will time out at 1147.309091s 2024-03-29T11:30:39.937 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.943+0000 7f4f38830640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f4f340ac5a0 0x7f4f34151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:40177/0 (socket says 172.21.15.135:40177) 2024-03-29T11:30:39.937 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.944+0000 7f4f38830640 1 -- 172.21.15.135:0/868567304 learned_addr learned my addr 172.21.15.135:0/868567304 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:30:39.941 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.947+0000 7f4f32ffd640 1 -- 172.21.15.135:0/868567304 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1094953431 0 0) 0x7f4f340aecb0 con 0x7f4f3414a700 2024-03-29T11:30:39.941 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.947+0000 7f4f32ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:39.941 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.947+0000 7f4f32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:39.941 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.947+0000 7f4f32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:39.941 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.947+0000 7f4f32ffd640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:30:39.941 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.947+0000 7f4f32ffd640 1 -- 172.21.15.135:0/868567304 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f1c003520 con 0x7f4f3414a700 2024-03-29T11:30:39.941 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.947+0000 7f4f32ffd640 1 -- 172.21.15.135:0/868567304 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2625656904 0 0) 0x7f4f340aba70 con 0x7f4f340ac5a0 2024-03-29T11:30:39.941 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.947+0000 7f4f32ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:39.941 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.947+0000 7f4f32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:39.941 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.947+0000 7f4f32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:39.941 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.947+0000 7f4f32ffd640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:30:39.942 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.947+0000 7f4f32ffd640 1 -- 172.21.15.135:0/868567304 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f340aecb0 con 0x7f4f340ac5a0 2024-03-29T11:30:39.942 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.947+0000 7f4f32ffd640 1 -- 172.21.15.135:0/868567304 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2646276465 0 0) 0x7f4f340af150 con 0x7f4f340b0770 2024-03-29T11:30:39.942 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.947+0000 7f4f32ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:39.942 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.947+0000 7f4f32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:39.942 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.947+0000 7f4f32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:39.942 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.947+0000 7f4f32ffd640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:30:39.942 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.947+0000 7f4f32ffd640 1 -- 172.21.15.135:0/868567304 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f340aba70 con 0x7f4f340b0770 2024-03-29T11:30:39.943 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.947+0000 7f4f32ffd640 1 -- 172.21.15.135:0/868567304 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3288782169 0 0) 0x7f4f340aba70 con 0x7f4f340b0770 2024-03-29T11:30:39.943 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.947+0000 7f4f32ffd640 1 -- 172.21.15.135:0/868567304 >> v1:172.21.15.194:6789/0 conn(0x7f4f3414a700 legacy=0x7f4f340abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:39.943 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.947+0000 7f4f32ffd640 1 -- 172.21.15.135:0/868567304 >> v1:172.21.15.135:6789/0 conn(0x7f4f340ac5a0 legacy=0x7f4f34151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:39.943 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.947+0000 7f4f32ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:30:39.943 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.947+0000 7f4f32ffd640 1 monclient: found mon.noname-b 2024-03-29T11:30:39.943 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.947+0000 7f4f32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:39.944 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.948+0000 7f4f32ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:30:39.944 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.948+0000 7f4f32ffd640 1 -- 172.21.15.135:0/868567304 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f340ac9d0 con 0x7f4f340b0770 2024-03-29T11:30:39.944 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.948+0000 7f4f32ffd640 10 monclient: _finish_auth 0 2024-03-29T11:30:39.944 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.948+0000 7f4f32ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:30:39.944 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.948+0000 7f4f32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:39.944 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.948+0000 7f4f32ffd640 1 -- 172.21.15.135:0/868567304 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f4f24001460 con 0x7f4f340b0770 2024-03-29T11:30:39.944 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.948+0000 7f4f32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:39.944 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.948+0000 7f4f32ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:39.944 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.948+0000 7f4f32ffd640 10 monclient: dump: 2024-03-29T11:30:39.944 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:39.944 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:39.944 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:39.944 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:39.944 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:39.944 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:39.944 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:39.945 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:39.945 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:39.945 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:39.945 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.948+0000 7f4f32ffd640 1 -- 172.21.15.135:0/868567304 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f24001680 con 0x7f4f340b0770 2024-03-29T11:30:39.945 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.948+0000 7f4f32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:39.945 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.948+0000 7f4f32ffd640 1 -- 172.21.15.135:0/868567304 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f4f24001a10 con 0x7f4f340b0770 2024-03-29T11:30:39.945 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.948+0000 7f4f32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:39.945 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.948+0000 7f4f32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:39.945 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.948+0000 7f4f32ffd640 10 monclient: dump: 2024-03-29T11:30:39.945 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:39.945 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:39.945 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:39.945 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:39.945 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:39.945 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:39.946 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:39.946 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:39.946 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:39.946 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:39.946 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.948+0000 7f4f3a293640 5 monclient: authenticate success, global_id 4457 2024-03-29T11:30:39.946 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.948+0000 7f4f3a293640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:39.946 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.948+0000 7f4f3a293640 4 set_mon_vals no callback set 2024-03-29T11:30:39.946 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.949+0000 7f4f3a293640 10 monclient: shutdown 2024-03-29T11:30:39.946 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.949+0000 7f4f3a293640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:39.946 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.949+0000 7f4f3a293640 1 -- 172.21.15.135:0/868567304 >> v1:172.21.15.135:6790/0 conn(0x7f4f340b0770 legacy=0x7f4f34149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:39.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.952+0000 7f4f3a293640 1 -- 172.21.15.135:0/868567304 shutdown_connections 2024-03-29T11:30:39.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.952+0000 7f4f3a293640 1 -- 172.21.15.135:0/868567304 >> 172.21.15.135:0/868567304 conn(0x7f4f3406b660 msgr2=0x7f4f3406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:39.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.952+0000 7f4f3a293640 1 -- 172.21.15.135:0/868567304 shutdown_connections 2024-03-29T11:30:39.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.953+0000 7f4f3a293640 1 -- 172.21.15.135:0/868567304 wait complete. 2024-03-29T11:30:39.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.953+0000 7f4f3a293640 10 monclient: build_initial_monmap 2024-03-29T11:30:39.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.953+0000 7f4f3a293640 10 monclient: monmap: 2024-03-29T11:30:39.948 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:30:39.948 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:39.948 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:30:39.948 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:30:39.948 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:39.948 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:39.948 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:30:39.948 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:30:39.949 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:30:39.949 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:39.949 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.954+0000 7f4f3a293640 1 librados: starting msgr at 2024-03-29T11:30:39.949 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.954+0000 7f4f3a293640 1 librados: starting objecter 2024-03-29T11:30:39.949 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.954+0000 7f4f3a293640 1 Processor -- start 2024-03-29T11:30:39.950 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.954+0000 7f4f3a293640 1 -- start start 2024-03-29T11:30:39.950 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.954+0000 7f4f3a293640 1 librados: setting wanted keys 2024-03-29T11:30:39.950 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.954+0000 7f4f3a293640 1 librados: calling monclient init 2024-03-29T11:30:39.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.954+0000 7f4f3a293640 10 monclient: init 2024-03-29T11:30:39.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.955+0000 7f4f3a293640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:39.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.955+0000 7f4f3a293640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:39.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.955+0000 7f4f3a293640 10 monclient(hunting): picked mon.noname-b con 0x7f4f340ac5a0 addr v1:172.21.15.194:6789/0 2024-03-29T11:30:39.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.955+0000 7f4f3a293640 10 monclient(hunting): picked mon.noname-a con 0x7f4f340b0770 addr v1:172.21.15.135:6789/0 2024-03-29T11:30:39.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.955+0000 7f4f3a293640 10 monclient(hunting): picked mon.noname-c con 0x7f4f3414a700 addr v1:172.21.15.135:6790/0 2024-03-29T11:30:39.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.955+0000 7f4f3a293640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f341263f0 con 0x7f4f340b0770 2024-03-29T11:30:39.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.955+0000 7f4f3a293640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f34145760 con 0x7f4f340ac5a0 2024-03-29T11:30:39.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.955+0000 7f4f3a293640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f341239d0 con 0x7f4f3414a700 2024-03-29T11:30:39.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.955+0000 7f4f3a293640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:39.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.955+0000 7f4f3a293640 10 monclient(hunting): authenticate will time out at 1147.324856s 2024-03-29T11:30:39.956 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.963+0000 7f4f337fe640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f4f340b0770 0x7f4f34144940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:40577/0 (socket says 172.21.15.135:40577) 2024-03-29T11:30:39.957 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.963+0000 7f4f337fe640 1 -- 172.21.15.135:0/591272753 learned_addr learned my addr 172.21.15.135:0/591272753 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:30:39.959 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.964+0000 7f4f30ff9640 10 client.?.objecter ms_handle_connect 0x7f4f340b0770 2024-03-29T11:30:39.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.964+0000 7f4f30ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:39.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.964+0000 7f4f30ff9640 1 -- 172.21.15.135:0/591272753 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 694586645 0 0) 0x7f4f341263f0 con 0x7f4f340b0770 2024-03-29T11:30:39.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.964+0000 7f4f30ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:30:39.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.964+0000 7f4f30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:39.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.964+0000 7f4f30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:39.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.964+0000 7f4f30ff9640 10 monclient(hunting): my global_id is 4446 2024-03-29T11:30:39.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.964+0000 7f4f30ff9640 1 -- 172.21.15.135:0/591272753 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f20004eb0 con 0x7f4f340b0770 2024-03-29T11:30:39.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 10 client.?.objecter ms_handle_connect 0x7f4f3414a700 2024-03-29T11:30:39.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:39.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 1 -- 172.21.15.135:0/591272753 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2893323040 0 0) 0x7f4f341239d0 con 0x7f4f3414a700 2024-03-29T11:30:39.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:30:39.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:39.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:39.961 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:30:39.961 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 1 -- 172.21.15.135:0/591272753 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f341263f0 con 0x7f4f3414a700 2024-03-29T11:30:39.961 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 10 client.?.objecter ms_handle_connect 0x7f4f340ac5a0 2024-03-29T11:30:39.961 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:39.961 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 1 -- 172.21.15.135:0/591272753 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3435146378 0 0) 0x7f4f20004eb0 con 0x7f4f340b0770 2024-03-29T11:30:39.961 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 1 -- 172.21.15.135:0/591272753 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f341239d0 con 0x7f4f340b0770 2024-03-29T11:30:39.961 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 1 -- 172.21.15.135:0/591272753 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2341296463 0 0) 0x7f4f341263f0 con 0x7f4f3414a700 2024-03-29T11:30:39.961 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 1 -- 172.21.15.135:0/591272753 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f20004eb0 con 0x7f4f3414a700 2024-03-29T11:30:39.961 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 1 -- 172.21.15.135:0/591272753 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f4f28001a10 con 0x7f4f340b0770 2024-03-29T11:30:39.961 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:39.961 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:39.962 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 10 monclient(hunting): dump: 2024-03-29T11:30:39.962 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:39.962 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:39.962 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:39.962 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:39.962 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:39.962 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:39.962 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:39.962 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:39.962 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:39.962 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:39.962 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:39.962 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 1 -- 172.21.15.135:0/591272753 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f4f2c001ae0 con 0x7f4f3414a700 2024-03-29T11:30:39.963 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:39.963 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:39.963 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 10 monclient(hunting): dump: 2024-03-29T11:30:39.963 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:39.963 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:39.963 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:39.963 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:39.963 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:39.963 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:39.963 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:39.963 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:39.963 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:39.963 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:39.963 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 1 -- 172.21.15.135:0/591272753 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2065407297 0 0) 0x7f4f20004eb0 con 0x7f4f3414a700 2024-03-29T11:30:39.963 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 1 -- 172.21.15.135:0/591272753 >> v1:172.21.15.194:6789/0 conn(0x7f4f340ac5a0 legacy=0x7f4f34126880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:39.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 1 -- 172.21.15.135:0/591272753 >> v1:172.21.15.135:6789/0 conn(0x7f4f340b0770 legacy=0x7f4f34144940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:39.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:30:39.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 1 monclient: found mon.c 2024-03-29T11:30:39.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:39.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:30:39.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 1 -- 172.21.15.135:0/591272753 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f34123b60 con 0x7f4f3414a700 2024-03-29T11:30:39.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 10 monclient: _finish_auth 0 2024-03-29T11:30:39.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:30:39.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:39.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f3a293640 5 monclient: authenticate success, global_id 4460 2024-03-29T11:30:39.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f3a293640 10 monclient: _renew_subs 2024-03-29T11:30:39.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f3a293640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:30:39.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.965+0000 7f4f3a293640 1 -- 172.21.15.135:0/591272753 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f34126ff0 con 0x7f4f3414a700 2024-03-29T11:30:39.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.966+0000 7f4f3a293640 10 client.4460.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:39.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.966+0000 7f4f3a293640 10 monclient: _renew_subs 2024-03-29T11:30:39.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.966+0000 7f4f3a293640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:30:39.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.966+0000 7f4f3a293640 1 -- 172.21.15.135:0/591272753 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f341df880 con 0x7f4f3414a700 2024-03-29T11:30:39.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.966+0000 7f4f3a293640 1 librados: init done 2024-03-29T11:30:39.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.967+0000 7f4f3a293640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:39.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.968+0000 7f4f30ff9640 1 -- 172.21.15.135:0/591272753 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f2c002010 con 0x7f4f3414a700 2024-03-29T11:30:39.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.968+0000 7f4f30ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:39.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.968+0000 7f4f327fc640 4 set_mon_vals no callback set 2024-03-29T11:30:39.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.969+0000 7f4f30ff9640 1 -- 172.21.15.135:0/591272753 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f4f2c004d00 con 0x7f4f3414a700 2024-03-29T11:30:39.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.969+0000 7f4f30ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:39.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.969+0000 7f4f30ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:39.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.969+0000 7f4f30ff9640 10 monclient: dump: 2024-03-29T11:30:39.967 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:39.967 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:39.967 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:39.968 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:39.968 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:39.968 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:39.968 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:39.968 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:39.968 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:39.968 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:39.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.969+0000 7f4f30ff9640 1 -- 172.21.15.135:0/591272753 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f4f2c004ee0 con 0x7f4f3414a700 2024-03-29T11:30:39.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.969+0000 7f4f30ff9640 1 -- 172.21.15.135:0/591272753 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f4f2c084d70 con 0x7f4f3414a700 2024-03-29T11:30:39.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.969+0000 7f4f30ff9640 10 client.4460.objecter ms_dispatch 0x7f4f34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:39.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.969+0000 7f4f30ff9640 3 client.4460.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:39.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.969+0000 7f4f30ff9640 3 client.4460.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:39.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.969+0000 7f4f30ff9640 20 client.4460.objecter dump_active .. 0 homeless 2024-03-29T11:30:39.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.970+0000 7f4f33fff640 10 monclient: get_auth_request con 0x7f4f2006dd40 auth_method 0 2024-03-29T11:30:39.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.971+0000 7f4f3a293640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:39.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.971+0000 7f4f3a293640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:30:39.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.971+0000 7f4f3a293640 1 -- 172.21.15.135:0/591272753 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f340ab560 con 0x7f4f3414a700 2024-03-29T11:30:39.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.974+0000 7f4f30ff9640 10 client.4460.objecter ms_handle_connect 0x7f4f2006dd40 2024-03-29T11:30:39.974 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.981+0000 7f4f30ff9640 1 -- 172.21.15.135:0/591272753 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4f2c0506d0 con 0x7f4f3414a700 2024-03-29T11:30:39.975 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.981+0000 7f4f30ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:39.975 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:39.981+0000 7f4f30ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:40.296 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:40.302+0000 7f4f3a293640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:40.296 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:40.302+0000 7f4f3a293640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:40.296 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:40.302+0000 7f4f3a293640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:30:40.296 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:40.302+0000 7f4f3a293640 1 -- 172.21.15.135:0/591272753 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f340b0b50 con 0x7f4f3414a700 2024-03-29T11:30:40.300 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:40.306+0000 7f4f30ff9640 1 -- 172.21.15.135:0/591272753 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f4f2c052600 con 0x7f4f3414a700 2024-03-29T11:30:40.300 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:40.306+0000 7f4f30ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:40.300 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:40.306+0000 7f4f30ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:40.300 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:30:40.300 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:40.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:40.311+0000 7f4f3a293640 10 librados: watch_flush enter 2024-03-29T11:30:40.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:40.311+0000 7f4f3a293640 10 librados: watch_flush exit 2024-03-29T11:30:40.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:40.311+0000 7f4f3a293640 20 client.4460.objecter shutdown clearing up homeless session... 2024-03-29T11:30:40.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:40.311+0000 7f4f3a293640 10 client.4460.objecter successfully canceled tick 2024-03-29T11:30:40.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:40.311+0000 7f4f3a293640 1 -- 172.21.15.135:0/591272753 >> v1:172.21.15.135:6816/3469875285 conn(0x7f4f2006dd40 legacy=0x7f4f20070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:40.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:40.311+0000 7f4f3a293640 10 monclient: shutdown 2024-03-29T11:30:40.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:40.311+0000 7f4f3a293640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:40.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:40.311+0000 7f4f3a293640 1 -- 172.21.15.135:0/591272753 >> v1:172.21.15.135:6790/0 conn(0x7f4f3414a700 legacy=0x7f4f34145050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:40.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:40.313+0000 7f4f3a293640 1 -- 172.21.15.135:0/591272753 shutdown_connections 2024-03-29T11:30:40.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:40.313+0000 7f4f3a293640 1 -- 172.21.15.135:0/591272753 >> 172.21.15.135:0/591272753 conn(0x7f4f3406b660 msgr2=0x7f4f340abe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:40.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:40.315+0000 7f4f3a293640 1 -- 172.21.15.135:0/591272753 shutdown_connections 2024-03-29T11:30:40.310 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:40.316+0000 7f4f3a293640 1 -- 172.21.15.135:0/591272753 wait complete. 2024-03-29T11:30:40.310 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:40.316+0000 7f4f3a293640 1 librados: shutdown 2024-03-29T11:30:40.327 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:47.134 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:47.296 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.303+0000 7efd5ac4e640 10 monclient: get_monmap_and_config 2024-03-29T11:30:47.296 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.303+0000 7efd5ac4e640 10 monclient: build_initial_monmap 2024-03-29T11:30:47.296 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.303+0000 7efd5ac4e640 10 monclient: monmap: 2024-03-29T11:30:47.296 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:30:47.296 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:47.296 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:30:47.304126+0000 2024-03-29T11:30:47.296 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:30:47.304126+0000 2024-03-29T11:30:47.297 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:47.297 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:47.297 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:30:47.297 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:30:47.297 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:30:47.297 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:47.300 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.304+0000 7efd5ac4e640 1 Processor -- start 2024-03-29T11:30:47.300 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.304+0000 7efd5ac4e640 1 -- start start 2024-03-29T11:30:47.300 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.304+0000 7efd5ac4e640 10 monclient: init 2024-03-29T11:30:47.300 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.305+0000 7efd5ac4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:47.301 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.305+0000 7efd5ac4e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:30:47.301 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.305+0000 7efd5ac4e640 10 monclient(hunting): picked mon.noname-b con 0x7efd54070170 addr v1:172.21.15.135:6790/0 2024-03-29T11:30:47.301 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.305+0000 7efd5ac4e640 10 monclient(hunting): picked mon.noname-c con 0x7efd540713f0 addr v1:172.21.15.194:6789/0 2024-03-29T11:30:47.301 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.305+0000 7efd5ac4e640 10 monclient(hunting): picked mon.noname-a con 0x7efd54070550 addr v1:172.21.15.135:6789/0 2024-03-29T11:30:47.301 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.305+0000 7efd5ac4e640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd5413dbd0 con 0x7efd54070550 2024-03-29T11:30:47.301 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.305+0000 7efd5ac4e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd5413ff70 con 0x7efd540713f0 2024-03-29T11:30:47.301 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.305+0000 7efd5ac4e640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd54140250 con 0x7efd54070170 2024-03-29T11:30:47.301 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.305+0000 7efd5ac4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:47.301 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.305+0000 7efd5ac4e640 10 monclient(hunting): authenticate will time out at 1154.673807s 2024-03-29T11:30:47.304 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.308+0000 7efd53fff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7efd540713f0 0x7efd541406e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:52996/0 (socket says 172.21.15.135:52996) 2024-03-29T11:30:47.304 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.308+0000 7efd53fff640 1 -- 172.21.15.135:0/2482703059 learned_addr learned my addr 172.21.15.135:0/2482703059 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:30:47.305 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.309+0000 7efd537fe640 1 -- 172.21.15.135:0/2482703059 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4131445205 0 0) 0x7efd5413ff70 con 0x7efd540713f0 2024-03-29T11:30:47.305 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.309+0000 7efd537fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:47.305 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.309+0000 7efd537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:47.305 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.309+0000 7efd537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:47.305 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.309+0000 7efd537fe640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:30:47.305 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.309+0000 7efd537fe640 1 -- 172.21.15.135:0/2482703059 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd3c003520 con 0x7efd540713f0 2024-03-29T11:30:47.305 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.310+0000 7efd537fe640 1 -- 172.21.15.135:0/2482703059 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1390421415 0 0) 0x7efd3c003520 con 0x7efd540713f0 2024-03-29T11:30:47.305 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.310+0000 7efd537fe640 1 -- 172.21.15.135:0/2482703059 >> v1:172.21.15.135:6790/0 conn(0x7efd54070170 legacy=0x7efd5413d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:30:47.305 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.310+0000 7efd537fe640 1 -- 172.21.15.135:0/2482703059 >> v1:172.21.15.135:6789/0 conn(0x7efd54070550 legacy=0x7efd5414b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:47.305 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.310+0000 7efd537fe640 10 monclient: _finish_hunting 0 2024-03-29T11:30:47.305 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.310+0000 7efd537fe640 1 monclient: found mon.noname-c 2024-03-29T11:30:47.305 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.310+0000 7efd537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:47.305 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.310+0000 7efd537fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:30:47.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.310+0000 7efd537fe640 1 -- 172.21.15.135:0/2482703059 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd54140df0 con 0x7efd540713f0 2024-03-29T11:30:47.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.310+0000 7efd537fe640 10 monclient: _finish_auth 0 2024-03-29T11:30:47.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.310+0000 7efd537fe640 10 monclient: _check_auth_tickets 2024-03-29T11:30:47.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.310+0000 7efd537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:47.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.310+0000 7efd537fe640 1 -- 172.21.15.135:0/2482703059 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7efd48001670 con 0x7efd540713f0 2024-03-29T11:30:47.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.310+0000 7efd537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:47.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.310+0000 7efd537fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:47.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.310+0000 7efd537fe640 10 monclient: dump: 2024-03-29T11:30:47.306 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:47.306 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:47.306 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:47.306 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:47.306 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:47.306 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:47.307 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:47.307 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:47.307 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:47.307 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:47.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.311+0000 7efd537fe640 1 -- 172.21.15.135:0/2482703059 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd48001940 con 0x7efd540713f0 2024-03-29T11:30:47.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.311+0000 7efd537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:47.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.311+0000 7efd537fe640 1 -- 172.21.15.135:0/2482703059 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7efd48001cd0 con 0x7efd540713f0 2024-03-29T11:30:47.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.311+0000 7efd537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:47.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.311+0000 7efd537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:47.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.311+0000 7efd537fe640 10 monclient: dump: 2024-03-29T11:30:47.307 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:47.307 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:47.307 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:47.307 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:47.307 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:47.307 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:47.308 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:47.308 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:47.308 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:47.308 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:47.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.311+0000 7efd5ac4e640 5 monclient: authenticate success, global_id 4459 2024-03-29T11:30:47.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.311+0000 7efd5ac4e640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:47.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.311+0000 7efd5ac4e640 4 set_mon_vals no callback set 2024-03-29T11:30:47.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.312+0000 7efd5ac4e640 10 monclient: shutdown 2024-03-29T11:30:47.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.312+0000 7efd5ac4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:47.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.312+0000 7efd5ac4e640 1 -- 172.21.15.135:0/2482703059 >> v1:172.21.15.194:6789/0 conn(0x7efd540713f0 legacy=0x7efd541406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:47.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.312+0000 7efd5ac4e640 1 -- 172.21.15.135:0/2482703059 shutdown_connections 2024-03-29T11:30:47.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.312+0000 7efd5ac4e640 1 -- 172.21.15.135:0/2482703059 >> 172.21.15.135:0/2482703059 conn(0x7efd54067610 msgr2=0x7efd540b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:47.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.312+0000 7efd5ac4e640 1 -- 172.21.15.135:0/2482703059 shutdown_connections 2024-03-29T11:30:47.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.312+0000 7efd5ac4e640 1 -- 172.21.15.135:0/2482703059 wait complete. 2024-03-29T11:30:47.312 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.313+0000 7efd5ac4e640 10 monclient: build_initial_monmap 2024-03-29T11:30:47.312 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.313+0000 7efd5ac4e640 10 monclient: monmap: 2024-03-29T11:30:47.312 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:30:47.312 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:47.312 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:30:47.312 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:30:47.312 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:47.312 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:47.313 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:30:47.313 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:30:47.313 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:30:47.313 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:47.313 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.313+0000 7efd5ac4e640 1 librados: starting msgr at 2024-03-29T11:30:47.313 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.313+0000 7efd5ac4e640 1 librados: starting objecter 2024-03-29T11:30:47.313 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.313+0000 7efd5ac4e640 1 Processor -- start 2024-03-29T11:30:47.313 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.313+0000 7efd5ac4e640 1 -- start start 2024-03-29T11:30:47.313 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.313+0000 7efd5ac4e640 1 librados: setting wanted keys 2024-03-29T11:30:47.313 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.313+0000 7efd5ac4e640 1 librados: calling monclient init 2024-03-29T11:30:47.313 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.313+0000 7efd5ac4e640 10 monclient: init 2024-03-29T11:30:47.313 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.314+0000 7efd5ac4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:47.313 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.314+0000 7efd5ac4e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:30:47.313 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.314+0000 7efd5ac4e640 10 monclient(hunting): picked mon.noname-c con 0x7efd54070170 addr v1:172.21.15.135:6790/0 2024-03-29T11:30:47.314 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.314+0000 7efd5ac4e640 10 monclient(hunting): picked mon.noname-a con 0x7efd54070550 addr v1:172.21.15.135:6789/0 2024-03-29T11:30:47.314 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.314+0000 7efd5ac4e640 10 monclient(hunting): picked mon.noname-b con 0x7efd540713f0 addr v1:172.21.15.194:6789/0 2024-03-29T11:30:47.314 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.314+0000 7efd5ac4e640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd541d6520 con 0x7efd54070550 2024-03-29T11:30:47.314 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.315+0000 7efd5ac4e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd541d73a0 con 0x7efd540713f0 2024-03-29T11:30:47.314 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.315+0000 7efd5ac4e640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd541d6770 con 0x7efd54070170 2024-03-29T11:30:47.314 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.315+0000 7efd5ac4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:47.314 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.315+0000 7efd5ac4e640 10 monclient(hunting): authenticate will time out at 1154.683081s 2024-03-29T11:30:47.314 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.315+0000 7efd591eb640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7efd540713f0 0x7efd541361f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:53282/0 (socket says 172.21.15.135:53282) 2024-03-29T11:30:47.314 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.315+0000 7efd591eb640 1 -- 172.21.15.135:0/1118951372 learned_addr learned my addr 172.21.15.135:0/1118951372 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:30:47.314 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.315+0000 7efd517fa640 10 client.?.objecter ms_handle_connect 0x7efd540713f0 2024-03-29T11:30:47.314 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.315+0000 7efd517fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:47.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.315+0000 7efd517fa640 1 -- 172.21.15.135:0/1118951372 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851636219 0 0) 0x7efd541d73a0 con 0x7efd540713f0 2024-03-29T11:30:47.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.315+0000 7efd517fa640 10 monclient(hunting): _negotiate 2024-03-29T11:30:47.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.315+0000 7efd517fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:47.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.315+0000 7efd517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:47.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.315+0000 7efd517fa640 10 monclient(hunting): my global_id is 4462 2024-03-29T11:30:47.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.315+0000 7efd517fa640 1 -- 172.21.15.135:0/1118951372 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd40004eb0 con 0x7efd540713f0 2024-03-29T11:30:47.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.316+0000 7efd517fa640 1 -- 172.21.15.135:0/1118951372 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 380502083 0 0) 0x7efd40004eb0 con 0x7efd540713f0 2024-03-29T11:30:47.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.316+0000 7efd517fa640 1 -- 172.21.15.135:0/1118951372 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd541d73a0 con 0x7efd540713f0 2024-03-29T11:30:47.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.316+0000 7efd517fa640 1 -- 172.21.15.135:0/1118951372 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7efd4c001ac0 con 0x7efd540713f0 2024-03-29T11:30:47.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.316+0000 7efd517fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:47.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.316+0000 7efd517fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:47.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.316+0000 7efd517fa640 10 monclient(hunting): dump: 2024-03-29T11:30:47.315 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:47.315 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:47.316 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:47.316 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:47.316 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:47.316 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:47.316 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:47.316 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:47.316 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:47.316 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:47.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.316+0000 7efd517fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:47.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.316+0000 7efd517fa640 1 -- 172.21.15.135:0/1118951372 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4287218702 0 0) 0x7efd541d73a0 con 0x7efd540713f0 2024-03-29T11:30:47.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.316+0000 7efd517fa640 1 -- 172.21.15.135:0/1118951372 >> v1:172.21.15.135:6790/0 conn(0x7efd54070170 legacy=0x7efd5406d430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:47.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.319+0000 7efd517fa640 1 -- 172.21.15.135:0/1118951372 >> v1:172.21.15.135:6789/0 conn(0x7efd54070550 legacy=0x7efd541d6c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:47.319 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.320+0000 7efd517fa640 10 monclient: _finish_hunting 0 2024-03-29T11:30:47.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.320+0000 7efd517fa640 1 monclient: found mon.b 2024-03-29T11:30:47.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.320+0000 7efd517fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:47.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.320+0000 7efd517fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:30:47.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.320+0000 7efd517fa640 1 -- 172.21.15.135:0/1118951372 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5413af50 con 0x7efd540713f0 2024-03-29T11:30:47.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.320+0000 7efd517fa640 10 monclient: _finish_auth 0 2024-03-29T11:30:47.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.320+0000 7efd517fa640 10 monclient: _check_auth_tickets 2024-03-29T11:30:47.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.320+0000 7efd517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:47.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.320+0000 7efd5ac4e640 5 monclient: authenticate success, global_id 4462 2024-03-29T11:30:47.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.320+0000 7efd5ac4e640 10 monclient: _renew_subs 2024-03-29T11:30:47.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.320+0000 7efd5ac4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:30:47.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.320+0000 7efd5ac4e640 1 -- 172.21.15.135:0/1118951372 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd5413b200 con 0x7efd540713f0 2024-03-29T11:30:47.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.320+0000 7efd5ac4e640 10 client.4462.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:47.321 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.320+0000 7efd5ac4e640 10 monclient: _renew_subs 2024-03-29T11:30:47.321 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.320+0000 7efd5ac4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:30:47.321 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.320+0000 7efd5ac4e640 1 -- 172.21.15.135:0/1118951372 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd5413cee0 con 0x7efd540713f0 2024-03-29T11:30:47.321 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.320+0000 7efd5ac4e640 1 librados: init done 2024-03-29T11:30:47.321 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.321+0000 7efd517fa640 1 -- 172.21.15.135:0/1118951372 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd4c002070 con 0x7efd540713f0 2024-03-29T11:30:47.321 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.321+0000 7efd517fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:47.321 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.321+0000 7efd517fa640 1 -- 172.21.15.135:0/1118951372 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7efd4c003b30 con 0x7efd540713f0 2024-03-29T11:30:47.321 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.321+0000 7efd517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:47.321 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.321+0000 7efd517fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:47.321 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.321+0000 7efd517fa640 10 monclient: dump: 2024-03-29T11:30:47.321 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:47.321 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:47.321 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:47.321 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:47.321 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:47.322 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:47.322 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:47.322 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:47.322 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:47.322 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:47.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.322+0000 7efd52ffd640 4 set_mon_vals no callback set 2024-03-29T11:30:47.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.322+0000 7efd5ac4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:47.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.324+0000 7efd517fa640 1 -- 172.21.15.135:0/1118951372 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7efd4c003c90 con 0x7efd540713f0 2024-03-29T11:30:47.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.324+0000 7efd5ac4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:47.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.324+0000 7efd5ac4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:30:47.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.324+0000 7efd5ac4e640 1 -- 172.21.15.135:0/1118951372 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd34005080 con 0x7efd540713f0 2024-03-29T11:30:47.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.325+0000 7efd517fa640 1 -- 172.21.15.135:0/1118951372 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7efd4c084c70 con 0x7efd540713f0 2024-03-29T11:30:47.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.325+0000 7efd517fa640 10 client.4462.objecter ms_dispatch 0x7efd54000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:47.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.325+0000 7efd517fa640 3 client.4462.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:47.323 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.325+0000 7efd517fa640 3 client.4462.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:47.323 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.325+0000 7efd517fa640 20 client.4462.objecter dump_active .. 0 homeless 2024-03-29T11:30:47.323 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.326+0000 7efd589ea640 10 monclient: get_auth_request con 0x7efd4006dbe0 auth_method 0 2024-03-29T11:30:47.323 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.326+0000 7efd517fa640 10 client.4462.objecter ms_handle_connect 0x7efd4006dbe0 2024-03-29T11:30:47.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.332+0000 7efd517fa640 1 -- 172.21.15.135:0/1118951372 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efd4c050550 con 0x7efd540713f0 2024-03-29T11:30:47.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.332+0000 7efd517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:47.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.332+0000 7efd517fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:47.638 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.645+0000 7efd5ac4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:47.638 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.645+0000 7efd5ac4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:47.638 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.645+0000 7efd5ac4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:30:47.638 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.645+0000 7efd5ac4e640 1 -- 172.21.15.135:0/1118951372 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd34005330 con 0x7efd540713f0 2024-03-29T11:30:47.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.648+0000 7efd517fa640 1 -- 172.21.15.135:0/1118951372 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7efd4c052480 con 0x7efd540713f0 2024-03-29T11:30:47.641 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.648+0000 7efd517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:47.641 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.648+0000 7efd517fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:47.641 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:30:47.641 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:47.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.651+0000 7efd5ac4e640 10 librados: watch_flush enter 2024-03-29T11:30:47.646 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.654+0000 7efd5ac4e640 10 librados: watch_flush exit 2024-03-29T11:30:47.646 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.654+0000 7efd5ac4e640 20 client.4462.objecter shutdown clearing up homeless session... 2024-03-29T11:30:47.646 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.654+0000 7efd5ac4e640 10 client.4462.objecter successfully canceled tick 2024-03-29T11:30:47.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.654+0000 7efd5ac4e640 1 -- 172.21.15.135:0/1118951372 >> v1:172.21.15.135:6816/3469875285 conn(0x7efd4006dbe0 legacy=0x7efd400700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:47.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.654+0000 7efd5ac4e640 10 monclient: shutdown 2024-03-29T11:30:47.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.654+0000 7efd5ac4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:47.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.654+0000 7efd5ac4e640 1 -- 172.21.15.135:0/1118951372 >> v1:172.21.15.194:6789/0 conn(0x7efd540713f0 legacy=0x7efd541361f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:47.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.655+0000 7efd5ac4e640 1 -- 172.21.15.135:0/1118951372 shutdown_connections 2024-03-29T11:30:47.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.655+0000 7efd5ac4e640 1 -- 172.21.15.135:0/1118951372 >> 172.21.15.135:0/1118951372 conn(0x7efd54067610 msgr2=0x7efd5413d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:47.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.655+0000 7efd5ac4e640 1 -- 172.21.15.135:0/1118951372 shutdown_connections 2024-03-29T11:30:47.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.655+0000 7efd5ac4e640 1 -- 172.21.15.135:0/1118951372 wait complete. 2024-03-29T11:30:47.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:47.655+0000 7efd5ac4e640 1 librados: shutdown 2024-03-29T11:30:47.667 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:54.475 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:54.637 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.644+0000 7fafe6948640 10 monclient: get_monmap_and_config 2024-03-29T11:30:54.637 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.644+0000 7fafe6948640 10 monclient: build_initial_monmap 2024-03-29T11:30:54.637 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.644+0000 7fafe6948640 10 monclient: monmap: 2024-03-29T11:30:54.637 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:30:54.637 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:54.637 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:30:54.644746+0000 2024-03-29T11:30:54.637 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:30:54.644746+0000 2024-03-29T11:30:54.637 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:54.637 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:54.637 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:30:54.638 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:30:54.638 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:30:54.638 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:54.638 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.645+0000 7fafe6948640 1 Processor -- start 2024-03-29T11:30:54.638 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.645+0000 7fafe6948640 1 -- start start 2024-03-29T11:30:54.639 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.645+0000 7fafe6948640 10 monclient: init 2024-03-29T11:30:54.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.646+0000 7fafe6948640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:54.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.646+0000 7fafe6948640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:30:54.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.646+0000 7fafe6948640 10 monclient(hunting): picked mon.noname-b con 0x7fafe01422b0 addr v1:172.21.15.135:6790/0 2024-03-29T11:30:54.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.647+0000 7fafe6948640 10 monclient(hunting): picked mon.noname-c con 0x7fafe0142690 addr v1:172.21.15.194:6789/0 2024-03-29T11:30:54.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.647+0000 7fafe6948640 10 monclient(hunting): picked mon.noname-a con 0x7fafe00b17d0 addr v1:172.21.15.135:6789/0 2024-03-29T11:30:54.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.647+0000 7fafe6948640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe0142c80 con 0x7fafe00b17d0 2024-03-29T11:30:54.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.647+0000 7fafe6948640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe00aae40 con 0x7fafe0142690 2024-03-29T11:30:54.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.647+0000 7fafe6948640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe013fbd0 con 0x7fafe01422b0 2024-03-29T11:30:54.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.647+0000 7fafe6948640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:54.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.647+0000 7fafe6948640 10 monclient(hunting): authenticate will time out at 1162.015147s 2024-03-29T11:30:54.641 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.647+0000 7fafdf7fe640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fafe0142690 0x7fafe00b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:42994/0 (socket says 172.21.15.135:42994) 2024-03-29T11:30:54.641 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.647+0000 7fafe4ee5640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fafe00b17d0 0x7fafe00b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:48768/0 (socket says 172.21.15.135:48768) 2024-03-29T11:30:54.641 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.647+0000 7fafe4ee5640 1 -- 172.21.15.135:0/1074771867 learned_addr learned my addr 172.21.15.135:0/1074771867 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:30:54.642 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 1 -- 172.21.15.135:0/1074771867 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622494064 0 0) 0x7fafe0142c80 con 0x7fafe00b17d0 2024-03-29T11:30:54.642 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:54.642 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:54.642 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:54.642 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 10 monclient(hunting): my global_id is 4452 2024-03-29T11:30:54.642 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 1 -- 172.21.15.135:0/1074771867 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafc8003520 con 0x7fafe00b17d0 2024-03-29T11:30:54.642 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 1 -- 172.21.15.135:0/1074771867 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 523243667 0 0) 0x7fafe00aae40 con 0x7fafe0142690 2024-03-29T11:30:54.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:54.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:54.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:54.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:30:54.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 1 -- 172.21.15.135:0/1074771867 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafe0142c80 con 0x7fafe0142690 2024-03-29T11:30:54.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 1 -- 172.21.15.135:0/1074771867 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3904795600 0 0) 0x7fafe0142c80 con 0x7fafe0142690 2024-03-29T11:30:54.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 1 -- 172.21.15.135:0/1074771867 >> v1:172.21.15.135:6790/0 conn(0x7fafe01422b0 legacy=0x7fafe013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:54.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 1 -- 172.21.15.135:0/1074771867 >> v1:172.21.15.135:6789/0 conn(0x7fafe00b17d0 legacy=0x7fafe00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:54.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:30:54.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 1 monclient: found mon.noname-c 2024-03-29T11:30:54.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:54.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:30:54.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 1 -- 172.21.15.135:0/1074771867 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe0151940 con 0x7fafe0142690 2024-03-29T11:30:54.644 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 10 monclient: _finish_auth 0 2024-03-29T11:30:54.644 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:30:54.644 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:54.644 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 1 -- 172.21.15.135:0/1074771867 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fafd4001630 con 0x7fafe0142690 2024-03-29T11:30:54.644 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:54.644 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:54.644 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.648+0000 7fafdeffd640 10 monclient: dump: 2024-03-29T11:30:54.644 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:54.644 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:54.644 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:54.644 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:54.644 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:54.644 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:54.644 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:54.645 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:54.645 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:54.645 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:54.645 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.649+0000 7fafe6948640 5 monclient: authenticate success, global_id 4465 2024-03-29T11:30:54.645 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.649+0000 7fafe6948640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:54.645 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.649+0000 7fafdeffd640 1 -- 172.21.15.135:0/1074771867 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafd4001810 con 0x7fafe0142690 2024-03-29T11:30:54.645 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.649+0000 7fafdeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:54.645 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.649+0000 7fafe6948640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:54.645 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.649+0000 7fafe6948640 4 set_mon_vals no callback set 2024-03-29T11:30:54.645 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.649+0000 7fafdeffd640 1 -- 172.21.15.135:0/1074771867 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fafd4001ba0 con 0x7fafe0142690 2024-03-29T11:30:54.645 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.649+0000 7fafdeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:54.645 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.649+0000 7fafdeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:54.645 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.649+0000 7fafdeffd640 10 monclient: dump: 2024-03-29T11:30:54.645 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:54.646 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:54.646 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:54.646 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:54.646 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:54.646 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:54.646 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:54.646 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:54.646 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:54.646 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:54.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.650+0000 7fafe6948640 10 monclient: shutdown 2024-03-29T11:30:54.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.650+0000 7fafe6948640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:54.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.650+0000 7fafe6948640 1 -- 172.21.15.135:0/1074771867 >> v1:172.21.15.194:6789/0 conn(0x7fafe0142690 legacy=0x7fafe00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:54.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.651+0000 7fafe6948640 1 -- 172.21.15.135:0/1074771867 shutdown_connections 2024-03-29T11:30:54.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.651+0000 7fafe6948640 1 -- 172.21.15.135:0/1074771867 >> 172.21.15.135:0/1074771867 conn(0x7fafe0066a00 msgr2=0x7fafe0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:54.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.651+0000 7fafe6948640 1 -- 172.21.15.135:0/1074771867 shutdown_connections 2024-03-29T11:30:54.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.651+0000 7fafe6948640 1 -- 172.21.15.135:0/1074771867 wait complete. 2024-03-29T11:30:54.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.653+0000 7fafe6948640 10 monclient: build_initial_monmap 2024-03-29T11:30:54.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.653+0000 7fafe6948640 10 monclient: monmap: 2024-03-29T11:30:54.648 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:30:54.649 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:54.649 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:30:54.649 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:30:54.649 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:54.649 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:54.649 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:30:54.649 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:30:54.649 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:30:54.649 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:54.649 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.654+0000 7fafe6948640 1 librados: starting msgr at 2024-03-29T11:30:54.649 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.654+0000 7fafe6948640 1 librados: starting objecter 2024-03-29T11:30:54.649 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.654+0000 7fafe6948640 1 Processor -- start 2024-03-29T11:30:54.649 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.654+0000 7fafe6948640 1 -- start start 2024-03-29T11:30:54.649 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.654+0000 7fafe6948640 1 librados: setting wanted keys 2024-03-29T11:30:54.649 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.655+0000 7fafe6948640 1 librados: calling monclient init 2024-03-29T11:30:54.650 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.655+0000 7fafe6948640 10 monclient: init 2024-03-29T11:30:54.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.656+0000 7fafe6948640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:54.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.656+0000 7fafe6948640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:54.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.656+0000 7fafe6948640 10 monclient(hunting): picked mon.noname-a con 0x7fafe00b17d0 addr v1:172.21.15.135:6789/0 2024-03-29T11:30:54.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.656+0000 7fafe6948640 10 monclient(hunting): picked mon.noname-c con 0x7fafe01422b0 addr v1:172.21.15.135:6790/0 2024-03-29T11:30:54.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.656+0000 7fafdffff640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fafe00b17d0 0x7fafe0141370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:49110/0 (socket says 172.21.15.135:49110) 2024-03-29T11:30:54.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.656+0000 7fafdffff640 1 -- 172.21.15.135:0/2732067122 learned_addr learned my addr 172.21.15.135:0/2732067122 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:30:54.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.656+0000 7fafe6948640 10 monclient(hunting): picked mon.noname-b con 0x7fafe01503f0 addr v1:172.21.15.194:6789/0 2024-03-29T11:30:54.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.657+0000 7fafe6948640 1 -- 172.21.15.135:0/2732067122 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe0151090 con 0x7fafe00b17d0 2024-03-29T11:30:54.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.657+0000 7fafe6948640 1 -- 172.21.15.135:0/2732067122 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe00b7270 con 0x7fafe01503f0 2024-03-29T11:30:54.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.657+0000 7fafe6948640 1 -- 172.21.15.135:0/2732067122 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe00b7400 con 0x7fafe01422b0 2024-03-29T11:30:54.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.657+0000 7fafe6948640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:54.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.657+0000 7fafe6948640 10 monclient(hunting): authenticate will time out at 1162.025363s 2024-03-29T11:30:54.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.657+0000 7fafdcff9640 10 client.?.objecter ms_handle_connect 0x7fafe00b17d0 2024-03-29T11:30:54.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.657+0000 7fafdcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:54.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.657+0000 7fafdcff9640 10 client.?.objecter ms_handle_connect 0x7fafe01422b0 2024-03-29T11:30:54.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.657+0000 7fafdcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:54.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.657+0000 7fafdcff9640 1 -- 172.21.15.135:0/2732067122 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3959973041 0 0) 0x7fafe0151090 con 0x7fafe00b17d0 2024-03-29T11:30:54.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.657+0000 7fafdcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:30:54.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.657+0000 7fafdcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:54.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.657+0000 7fafdcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:54.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.657+0000 7fafdcff9640 10 monclient(hunting): my global_id is 4455 2024-03-29T11:30:54.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.657+0000 7fafdcff9640 1 -- 172.21.15.135:0/2732067122 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafcc004eb0 con 0x7fafe00b17d0 2024-03-29T11:30:54.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.657+0000 7fafdcff9640 10 client.?.objecter ms_handle_connect 0x7fafe01503f0 2024-03-29T11:30:54.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.657+0000 7fafdcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:54.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.658+0000 7fafdcff9640 1 -- 172.21.15.135:0/2732067122 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225982111 0 0) 0x7fafe00b7400 con 0x7fafe01422b0 2024-03-29T11:30:54.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.658+0000 7fafdcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:30:54.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.658+0000 7fafdcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:54.659 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.658+0000 7fafdcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:54.659 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.658+0000 7fafdcff9640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:30:54.659 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.658+0000 7fafdcff9640 1 -- 172.21.15.135:0/2732067122 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafe0151090 con 0x7fafe01422b0 2024-03-29T11:30:54.659 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.658+0000 7fafdcff9640 1 -- 172.21.15.135:0/2732067122 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1759952469 0 0) 0x7fafe00b7270 con 0x7fafe01503f0 2024-03-29T11:30:54.659 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.658+0000 7fafdcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:30:54.659 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.658+0000 7fafdcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:54.659 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.658+0000 7fafdcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:54.659 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.658+0000 7fafdcff9640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:30:54.659 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.658+0000 7fafdcff9640 1 -- 172.21.15.135:0/2732067122 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafe00b7400 con 0x7fafe01503f0 2024-03-29T11:30:54.659 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.658+0000 7fafdcff9640 1 -- 172.21.15.135:0/2732067122 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2272793119 0 0) 0x7fafe0151090 con 0x7fafe01422b0 2024-03-29T11:30:54.659 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.658+0000 7fafdcff9640 1 -- 172.21.15.135:0/2732067122 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafe00b7270 con 0x7fafe01422b0 2024-03-29T11:30:54.659 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.658+0000 7fafdcff9640 1 -- 172.21.15.135:0/2732067122 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fafd4001ba0 con 0x7fafe01422b0 2024-03-29T11:30:54.659 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.658+0000 7fafdcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:54.660 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.658+0000 7fafdcff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:54.660 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.658+0000 7fafdcff9640 10 monclient(hunting): dump: 2024-03-29T11:30:54.660 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:54.660 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:54.660 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:54.660 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:54.660 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:54.660 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:54.660 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:54.660 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:54.660 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:54.660 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:54.661 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.658+0000 7fafdcff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:54.661 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.658+0000 7fafdcff9640 1 -- 172.21.15.135:0/2732067122 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2822125920 0 0) 0x7fafcc004eb0 con 0x7fafe00b17d0 2024-03-29T11:30:54.661 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.658+0000 7fafdcff9640 1 -- 172.21.15.135:0/2732067122 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafe0151090 con 0x7fafe00b17d0 2024-03-29T11:30:54.661 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.658+0000 7fafdcff9640 1 -- 172.21.15.135:0/2732067122 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fafd0001990 con 0x7fafe00b17d0 2024-03-29T11:30:54.661 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.658+0000 7fafdcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:54.661 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.658+0000 7fafdcff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:54.661 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.658+0000 7fafdcff9640 10 monclient(hunting): dump: 2024-03-29T11:30:54.661 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:54.661 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:54.661 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:54.661 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:54.661 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:54.661 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:54.661 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:54.662 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:54.662 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:54.662 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:54.662 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.659+0000 7fafdcff9640 1 -- 172.21.15.135:0/2732067122 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1775797241 0 0) 0x7fafe00b7400 con 0x7fafe01503f0 2024-03-29T11:30:54.662 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.659+0000 7fafdcff9640 1 -- 172.21.15.135:0/2732067122 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafcc004eb0 con 0x7fafe01503f0 2024-03-29T11:30:54.662 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.659+0000 7fafdcff9640 1 -- 172.21.15.135:0/2732067122 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fafd80015b0 con 0x7fafe01503f0 2024-03-29T11:30:54.662 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.659+0000 7fafdcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:54.662 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.659+0000 7fafdcff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:54.662 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.659+0000 7fafdcff9640 10 monclient(hunting): dump: 2024-03-29T11:30:54.662 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:54.662 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:54.662 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:54.662 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:54.662 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:54.663 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:54.663 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:54.663 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:54.663 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:54.663 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:54.663 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.659+0000 7fafdcff9640 1 -- 172.21.15.135:0/2732067122 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1681094324 0 0) 0x7fafe00b7270 con 0x7fafe01422b0 2024-03-29T11:30:54.663 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.659+0000 7fafdcff9640 1 -- 172.21.15.135:0/2732067122 >> v1:172.21.15.194:6789/0 conn(0x7fafe01503f0 legacy=0x7fafe012cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:54.663 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.659+0000 7fafdcff9640 1 -- 172.21.15.135:0/2732067122 >> v1:172.21.15.135:6789/0 conn(0x7fafe00b17d0 legacy=0x7fafe0141370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:54.663 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.659+0000 7fafdcff9640 10 monclient: _finish_hunting 0 2024-03-29T11:30:54.663 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.659+0000 7fafdcff9640 1 monclient: found mon.c 2024-03-29T11:30:54.663 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.659+0000 7fafdcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:54.663 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.659+0000 7fafdcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:30:54.663 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.659+0000 7fafdcff9640 1 -- 172.21.15.135:0/2732067122 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe00b7590 con 0x7fafe01422b0 2024-03-29T11:30:54.663 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.659+0000 7fafdcff9640 10 monclient: _finish_auth 0 2024-03-29T11:30:54.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.659+0000 7fafdcff9640 10 monclient: _check_auth_tickets 2024-03-29T11:30:54.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.659+0000 7fafdcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:54.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.659+0000 7fafe6948640 5 monclient: authenticate success, global_id 4466 2024-03-29T11:30:54.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.659+0000 7fafe6948640 10 monclient: _renew_subs 2024-03-29T11:30:54.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.659+0000 7fafe6948640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:30:54.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.659+0000 7fafe6948640 1 -- 172.21.15.135:0/2732067122 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe012d4a0 con 0x7fafe01422b0 2024-03-29T11:30:54.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.659+0000 7fafe6948640 10 client.4466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:54.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.659+0000 7fafe6948640 10 monclient: _renew_subs 2024-03-29T11:30:54.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.659+0000 7fafe6948640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:30:54.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.659+0000 7fafe6948640 1 -- 172.21.15.135:0/2732067122 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe00b59f0 con 0x7fafe01422b0 2024-03-29T11:30:54.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.659+0000 7fafe6948640 1 librados: init done 2024-03-29T11:30:54.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.660+0000 7fafdcff9640 1 -- 172.21.15.135:0/2732067122 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafd4001110 con 0x7fafe01422b0 2024-03-29T11:30:54.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.660+0000 7fafdcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:54.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.660+0000 7fafde7fc640 4 set_mon_vals no callback set 2024-03-29T11:30:54.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.660+0000 7fafdcff9640 1 -- 172.21.15.135:0/2732067122 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fafd4004c30 con 0x7fafe01422b0 2024-03-29T11:30:54.665 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.660+0000 7fafdcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:54.665 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.660+0000 7fafdcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:54.665 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.660+0000 7fafdcff9640 10 monclient: dump: 2024-03-29T11:30:54.665 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:30:54.665 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:30:54.665 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:54.665 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:30:54.665 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:54.665 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:30:54.665 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:30:54.665 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:30:54.665 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:30:54.665 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:30:54.666 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.661+0000 7fafe6948640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:54.666 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.661+0000 7fafe6948640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:54.666 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.661+0000 7fafe6948640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:30:54.666 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.661+0000 7fafe6948640 1 -- 172.21.15.135:0/2732067122 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc0005080 con 0x7fafe01422b0 2024-03-29T11:30:54.666 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.663+0000 7fafdcff9640 1 -- 172.21.15.135:0/2732067122 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fafd401ac00 con 0x7fafe01422b0 2024-03-29T11:30:54.666 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.663+0000 7fafdcff9640 1 -- 172.21.15.135:0/2732067122 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7fafd40848b0 con 0x7fafe01422b0 2024-03-29T11:30:54.666 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.663+0000 7fafdcff9640 10 client.4466.objecter ms_dispatch 0x7fafe0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:54.666 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.663+0000 7fafdcff9640 3 client.4466.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:54.666 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.663+0000 7fafdcff9640 3 client.4466.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:54.666 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.663+0000 7fafdcff9640 20 client.4466.objecter dump_active .. 0 homeless 2024-03-29T11:30:54.666 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.664+0000 7fafdffff640 10 monclient: get_auth_request con 0x7fafcc06e170 auth_method 0 2024-03-29T11:30:54.666 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.664+0000 7fafdcff9640 10 client.4466.objecter ms_handle_connect 0x7fafcc06e170 2024-03-29T11:30:54.667 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.672+0000 7fafdcff9640 1 -- 172.21.15.135:0/2732067122 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fafd40501f0 con 0x7fafe01422b0 2024-03-29T11:30:54.667 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.672+0000 7fafdcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:54.667 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:54.672+0000 7fafdcff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:55.090 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:55.097+0000 7fafe6948640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:55.090 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:55.097+0000 7fafe6948640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:55.090 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:55.097+0000 7fafe6948640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:30:55.090 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:55.097+0000 7fafe6948640 1 -- 172.21.15.135:0/2732067122 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fafc0005330 con 0x7fafe01422b0 2024-03-29T11:30:55.091 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:55.099+0000 7fafdcff9640 1 -- 172.21.15.135:0/2732067122 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7fafd40501f0 con 0x7fafe01422b0 2024-03-29T11:30:55.092 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:55.099+0000 7fafdcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:55.092 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:55.099+0000 7fafdcff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:55.092 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:30:55.092 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:55.096 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:55.103+0000 7fafe6948640 10 librados: watch_flush enter 2024-03-29T11:30:55.096 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:55.103+0000 7fafe6948640 10 librados: watch_flush exit 2024-03-29T11:30:55.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:55.103+0000 7fafe6948640 20 client.4466.objecter shutdown clearing up homeless session... 2024-03-29T11:30:55.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:55.103+0000 7fafe6948640 10 client.4466.objecter successfully canceled tick 2024-03-29T11:30:55.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:55.103+0000 7fafe6948640 1 -- 172.21.15.135:0/2732067122 >> v1:172.21.15.135:6816/3469875285 conn(0x7fafcc06e170 legacy=0x7fafcc070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:55.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:55.103+0000 7fafe6948640 10 monclient: shutdown 2024-03-29T11:30:55.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:55.103+0000 7fafe6948640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:55.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:55.103+0000 7fafe6948640 1 -- 172.21.15.135:0/2732067122 >> v1:172.21.15.135:6790/0 conn(0x7fafe01422b0 legacy=0x7fafe01daba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:55.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:55.103+0000 7fafe6948640 1 -- 172.21.15.135:0/2732067122 shutdown_connections 2024-03-29T11:30:55.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:55.104+0000 7fafe6948640 1 -- 172.21.15.135:0/2732067122 >> 172.21.15.135:0/2732067122 conn(0x7fafe0066a00 msgr2=0x7fafe013d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:55.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:55.104+0000 7fafe6948640 1 -- 172.21.15.135:0/2732067122 shutdown_connections 2024-03-29T11:30:55.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:55.104+0000 7fafe6948640 1 -- 172.21.15.135:0/2732067122 wait complete. 2024-03-29T11:30:55.098 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:30:55.104+0000 7fafe6948640 1 librados: shutdown 2024-03-29T11:30:55.113 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:01.921 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:02.068 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.075+0000 7fd4e8a1e640 10 monclient: get_monmap_and_config 2024-03-29T11:31:02.068 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.075+0000 7fd4e8a1e640 10 monclient: build_initial_monmap 2024-03-29T11:31:02.068 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.075+0000 7fd4e8a1e640 10 monclient: monmap: 2024-03-29T11:31:02.069 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:31:02.069 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:02.069 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:31:02.075957+0000 2024-03-29T11:31:02.069 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:31:02.075957+0000 2024-03-29T11:31:02.069 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:02.069 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:02.069 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:31:02.069 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:31:02.069 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:31:02.069 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:02.070 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.076+0000 7fd4e8a1e640 1 Processor -- start 2024-03-29T11:31:02.070 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.076+0000 7fd4e8a1e640 1 -- start start 2024-03-29T11:31:02.070 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.076+0000 7fd4e8a1e640 10 monclient: init 2024-03-29T11:31:02.070 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.077+0000 7fd4e8a1e640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:02.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.077+0000 7fd4e8a1e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:31:02.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.077+0000 7fd4e8a1e640 10 monclient(hunting): picked mon.noname-b con 0x7fd4e41457f0 addr v1:172.21.15.135:6790/0 2024-03-29T11:31:02.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.077+0000 7fd4e8a1e640 10 monclient(hunting): picked mon.noname-a con 0x7fd4e413f650 addr v1:172.21.15.135:6789/0 2024-03-29T11:31:02.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.077+0000 7fd4e8a1e640 10 monclient(hunting): picked mon.noname-c con 0x7fd4e40a9d70 addr v1:172.21.15.194:6789/0 2024-03-29T11:31:02.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.077+0000 7fd4e8a1e640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e40aaef0 con 0x7fd4e413f650 2024-03-29T11:31:02.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.077+0000 7fd4e8a1e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e4145bd0 con 0x7fd4e40a9d70 2024-03-29T11:31:02.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.077+0000 7fd4e8a1e640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e4145eb0 con 0x7fd4e41457f0 2024-03-29T11:31:02.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.077+0000 7fd4e8a1e640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:02.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.077+0000 7fd4e8a1e640 10 monclient(hunting): authenticate will time out at 1169.445638s 2024-03-29T11:31:02.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.077+0000 7fd4e2d9d640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fd4e40a9d70 0x7fd4e414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:35730/0 (socket says 172.21.15.135:35730) 2024-03-29T11:31:02.071 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.077+0000 7fd4e2d9d640 1 -- 172.21.15.135:0/2952542611 learned_addr learned my addr 172.21.15.135:0/2952542611 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:31:02.074 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.078+0000 7fd4e159a640 1 -- 172.21.15.135:0/2952542611 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 737824393 0 0) 0x7fd4e4145bd0 con 0x7fd4e40a9d70 2024-03-29T11:31:02.074 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.078+0000 7fd4e159a640 10 monclient(hunting): _negotiate 2024-03-29T11:31:02.074 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.078+0000 7fd4e159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:02.074 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.078+0000 7fd4e159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:02.074 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.078+0000 7fd4e159a640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:31:02.074 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 1 -- 172.21.15.135:0/2952542611 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4cc003520 con 0x7fd4e40a9d70 2024-03-29T11:31:02.074 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 1 -- 172.21.15.135:0/2952542611 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186417175 0 0) 0x7fd4e40aaef0 con 0x7fd4e413f650 2024-03-29T11:31:02.074 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 10 monclient(hunting): _negotiate 2024-03-29T11:31:02.074 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:02.074 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:02.074 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:31:02.075 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 1 -- 172.21.15.135:0/2952542611 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4e4145bd0 con 0x7fd4e413f650 2024-03-29T11:31:02.075 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 1 -- 172.21.15.135:0/2952542611 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3631366944 0 0) 0x7fd4e4145eb0 con 0x7fd4e41457f0 2024-03-29T11:31:02.075 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 10 monclient(hunting): _negotiate 2024-03-29T11:31:02.075 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:02.075 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:02.075 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:31:02.075 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 1 -- 172.21.15.135:0/2952542611 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4e40aaef0 con 0x7fd4e41457f0 2024-03-29T11:31:02.075 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 1 -- 172.21.15.135:0/2952542611 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 736122331 0 0) 0x7fd4cc003520 con 0x7fd4e40a9d70 2024-03-29T11:31:02.075 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 1 -- 172.21.15.135:0/2952542611 >> v1:172.21.15.135:6790/0 conn(0x7fd4e41457f0 legacy=0x7fd4e413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:02.075 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 1 -- 172.21.15.135:0/2952542611 >> v1:172.21.15.135:6789/0 conn(0x7fd4e413f650 legacy=0x7fd4e40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:02.075 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 10 monclient: _finish_hunting 0 2024-03-29T11:31:02.075 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 1 monclient: found mon.noname-c 2024-03-29T11:31:02.075 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:02.075 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:31:02.076 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 1 -- 172.21.15.135:0/2952542611 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e4146340 con 0x7fd4e40a9d70 2024-03-29T11:31:02.076 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 10 monclient: _finish_auth 0 2024-03-29T11:31:02.076 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 10 monclient: _check_auth_tickets 2024-03-29T11:31:02.076 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:02.076 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 1 -- 172.21.15.135:0/2952542611 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fd4d8001690 con 0x7fd4e40a9d70 2024-03-29T11:31:02.076 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:02.076 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:02.076 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e159a640 10 monclient: dump: 2024-03-29T11:31:02.076 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:02.076 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:02.076 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:02.076 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:02.076 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:02.076 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:02.077 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:02.077 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:02.077 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:02.077 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:02.077 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e8a1e640 5 monclient: authenticate success, global_id 4471 2024-03-29T11:31:02.077 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.079+0000 7fd4e8a1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:02.077 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.080+0000 7fd4e159a640 1 -- 172.21.15.135:0/2952542611 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4d80018b0 con 0x7fd4e40a9d70 2024-03-29T11:31:02.077 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.080+0000 7fd4e159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:02.077 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.080+0000 7fd4e159a640 1 -- 172.21.15.135:0/2952542611 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fd4d8001c40 con 0x7fd4e40a9d70 2024-03-29T11:31:02.077 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.080+0000 7fd4e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:02.077 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.080+0000 7fd4e159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:02.077 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.080+0000 7fd4e159a640 10 monclient: dump: 2024-03-29T11:31:02.077 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:02.077 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:02.077 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:02.078 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:02.078 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:02.078 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:02.078 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:02.078 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:02.078 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:02.078 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:02.078 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.080+0000 7fd4e8a1e640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:02.078 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.080+0000 7fd4e8a1e640 4 set_mon_vals no callback set 2024-03-29T11:31:02.078 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.081+0000 7fd4e8a1e640 10 monclient: shutdown 2024-03-29T11:31:02.078 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.081+0000 7fd4e8a1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:02.078 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.081+0000 7fd4e8a1e640 1 -- 172.21.15.135:0/2952542611 >> v1:172.21.15.194:6789/0 conn(0x7fd4e40a9d70 legacy=0x7fd4e414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:02.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.081+0000 7fd4e8a1e640 1 -- 172.21.15.135:0/2952542611 shutdown_connections 2024-03-29T11:31:02.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.081+0000 7fd4e8a1e640 1 -- 172.21.15.135:0/2952542611 >> 172.21.15.135:0/2952542611 conn(0x7fd4e4066a00 msgr2=0x7fd4e4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:02.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.081+0000 7fd4e8a1e640 1 -- 172.21.15.135:0/2952542611 shutdown_connections 2024-03-29T11:31:02.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.081+0000 7fd4e8a1e640 1 -- 172.21.15.135:0/2952542611 wait complete. 2024-03-29T11:31:02.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.082+0000 7fd4e8a1e640 10 monclient: build_initial_monmap 2024-03-29T11:31:02.098 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.082+0000 7fd4e8a1e640 10 monclient: monmap: 2024-03-29T11:31:02.098 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:31:02.098 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:02.098 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:31:02.098 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:31:02.098 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:02.098 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:02.098 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:31:02.099 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:31:02.099 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:31:02.099 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:02.099 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.083+0000 7fd4e8a1e640 1 librados: starting msgr at 2024-03-29T11:31:02.099 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.083+0000 7fd4e8a1e640 1 librados: starting objecter 2024-03-29T11:31:02.099 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.083+0000 7fd4e8a1e640 1 Processor -- start 2024-03-29T11:31:02.099 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.083+0000 7fd4e8a1e640 1 -- start start 2024-03-29T11:31:02.099 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.083+0000 7fd4e8a1e640 1 librados: setting wanted keys 2024-03-29T11:31:02.099 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.083+0000 7fd4e8a1e640 1 librados: calling monclient init 2024-03-29T11:31:02.099 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.083+0000 7fd4e8a1e640 10 monclient: init 2024-03-29T11:31:02.100 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.084+0000 7fd4e8a1e640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:02.100 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.084+0000 7fd4e8a1e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:31:02.100 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.084+0000 7fd4e8a1e640 10 monclient(hunting): picked mon.noname-a con 0x7fd4e40a9d70 addr v1:172.21.15.135:6789/0 2024-03-29T11:31:02.100 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.084+0000 7fd4e8a1e640 10 monclient(hunting): picked mon.noname-b con 0x7fd4e413f650 addr v1:172.21.15.194:6789/0 2024-03-29T11:31:02.100 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.084+0000 7fd4e8a1e640 10 monclient(hunting): picked mon.noname-c con 0x7fd4e41457f0 addr v1:172.21.15.135:6790/0 2024-03-29T11:31:02.100 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.084+0000 7fd4e8a1e640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e4138cc0 con 0x7fd4e40a9d70 2024-03-29T11:31:02.100 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.084+0000 7fd4e8a1e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e4138820 con 0x7fd4e413f650 2024-03-29T11:31:02.100 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.084+0000 7fd4e8a1e640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e406fa00 con 0x7fd4e41457f0 2024-03-29T11:31:02.100 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.084+0000 7fd4e8a1e640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:02.101 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.084+0000 7fd4e8a1e640 10 monclient(hunting): authenticate will time out at 1169.452407s 2024-03-29T11:31:02.101 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.084+0000 7fd4e1d9b640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fd4e413f650 0x7fd4e4138110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:36010/0 (socket says 172.21.15.135:36010) 2024-03-29T11:31:02.101 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.084+0000 7fd4e2d9d640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7fd4e41457f0 0x7fd4e41def60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:57876/0 (socket says 172.21.15.135:57876) 2024-03-29T11:31:02.101 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.084+0000 7fd4e1d9b640 1 -- 172.21.15.135:0/1218828220 learned_addr learned my addr 172.21.15.135:0/1218828220 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:31:02.101 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.084+0000 7fd4d2ffd640 10 client.?.objecter ms_handle_connect 0x7fd4e413f650 2024-03-29T11:31:02.101 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.084+0000 7fd4d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:02.101 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.084+0000 7fd4d2ffd640 10 client.?.objecter ms_handle_connect 0x7fd4e41457f0 2024-03-29T11:31:02.101 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.084+0000 7fd4d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:02.101 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.084+0000 7fd4d2ffd640 1 -- 172.21.15.135:0/1218828220 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4056634787 0 0) 0x7fd4e406fa00 con 0x7fd4e41457f0 2024-03-29T11:31:02.101 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.084+0000 7fd4d2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:02.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.084+0000 7fd4d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:02.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.084+0000 7fd4d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:02.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.084+0000 7fd4d2ffd640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:31:02.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.084+0000 7fd4d2ffd640 1 -- 172.21.15.135:0/1218828220 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4c4004eb0 con 0x7fd4e41457f0 2024-03-29T11:31:02.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.085+0000 7fd4d2ffd640 1 -- 172.21.15.135:0/1218828220 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136040049 0 0) 0x7fd4e4138820 con 0x7fd4e413f650 2024-03-29T11:31:02.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.085+0000 7fd4d2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:02.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.085+0000 7fd4d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:02.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.085+0000 7fd4d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:02.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.085+0000 7fd4d2ffd640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:31:02.103 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.085+0000 7fd4d2ffd640 1 -- 172.21.15.135:0/1218828220 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4e406fa00 con 0x7fd4e413f650 2024-03-29T11:31:02.103 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.085+0000 7fd4d2ffd640 10 client.?.objecter ms_handle_connect 0x7fd4e40a9d70 2024-03-29T11:31:02.103 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.085+0000 7fd4d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:02.103 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.085+0000 7fd4d2ffd640 1 -- 172.21.15.135:0/1218828220 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2254964655 0 0) 0x7fd4e4138cc0 con 0x7fd4e40a9d70 2024-03-29T11:31:02.103 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.085+0000 7fd4d2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:02.103 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.085+0000 7fd4d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:02.103 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.085+0000 7fd4d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:02.103 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.085+0000 7fd4d2ffd640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:31:02.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.085+0000 7fd4d2ffd640 1 -- 172.21.15.135:0/1218828220 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4e4138820 con 0x7fd4e40a9d70 2024-03-29T11:31:02.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.085+0000 7fd4d2ffd640 1 -- 172.21.15.135:0/1218828220 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2989611138 0 0) 0x7fd4e406fa00 con 0x7fd4e413f650 2024-03-29T11:31:02.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.085+0000 7fd4d2ffd640 1 -- 172.21.15.135:0/1218828220 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4e4138cc0 con 0x7fd4e413f650 2024-03-29T11:31:02.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.085+0000 7fd4d2ffd640 1 -- 172.21.15.135:0/1218828220 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fd4d40018a0 con 0x7fd4e413f650 2024-03-29T11:31:02.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.085+0000 7fd4d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:02.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.085+0000 7fd4d2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:02.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.085+0000 7fd4d2ffd640 10 monclient(hunting): dump: 2024-03-29T11:31:02.105 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:02.105 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:02.105 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:02.105 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:02.105 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:02.105 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:02.105 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:02.105 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:02.105 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:02.106 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:02.106 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.085+0000 7fd4d2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:02.106 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.085+0000 7fd4d2ffd640 1 -- 172.21.15.135:0/1218828220 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1959977765 0 0) 0x7fd4c4004eb0 con 0x7fd4e41457f0 2024-03-29T11:31:02.106 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.085+0000 7fd4d2ffd640 1 -- 172.21.15.135:0/1218828220 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4e406fa00 con 0x7fd4e41457f0 2024-03-29T11:31:02.106 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.085+0000 7fd4d2ffd640 1 -- 172.21.15.135:0/1218828220 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fd4d80015d0 con 0x7fd4e41457f0 2024-03-29T11:31:02.106 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.085+0000 7fd4d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:02.106 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4d2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:02.107 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4d2ffd640 10 monclient(hunting): dump: 2024-03-29T11:31:02.107 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:02.107 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:02.107 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:02.107 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:02.107 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:02.107 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:02.108 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:02.108 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:02.108 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:02.108 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:02.108 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4d2ffd640 1 -- 172.21.15.135:0/1218828220 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1033281068 0 0) 0x7fd4e406fa00 con 0x7fd4e41457f0 2024-03-29T11:31:02.108 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4d2ffd640 1 -- 172.21.15.135:0/1218828220 >> v1:172.21.15.194:6789/0 conn(0x7fd4e413f650 legacy=0x7fd4e4138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:02.108 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4d2ffd640 1 -- 172.21.15.135:0/1218828220 >> v1:172.21.15.135:6789/0 conn(0x7fd4e40a9d70 legacy=0x7fd4e413b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:02.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4d2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:31:02.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4d2ffd640 1 monclient: found mon.c 2024-03-29T11:31:02.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:02.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4d2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:02.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4d2ffd640 1 -- 172.21.15.135:0/1218828220 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e406fce0 con 0x7fd4e41457f0 2024-03-29T11:31:02.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4d2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:31:02.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4d2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:31:02.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:02.110 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4e8a1e640 5 monclient: authenticate success, global_id 4472 2024-03-29T11:31:02.110 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4e8a1e640 10 monclient: _renew_subs 2024-03-29T11:31:02.110 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4e8a1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:02.110 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4e8a1e640 1 -- 172.21.15.135:0/1218828220 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e4071e20 con 0x7fd4e41457f0 2024-03-29T11:31:02.110 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4d2ffd640 1 -- 172.21.15.135:0/1218828220 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4d8004a90 con 0x7fd4e41457f0 2024-03-29T11:31:02.110 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:02.110 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4d2ffd640 1 -- 172.21.15.135:0/1218828220 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fd4d8004da0 con 0x7fd4e41457f0 2024-03-29T11:31:02.111 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:02.111 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:02.111 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4d2ffd640 10 monclient: dump: 2024-03-29T11:31:02.111 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:02.111 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:02.111 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:02.111 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:02.111 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:02.112 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:02.112 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:02.112 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:02.112 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:02.112 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:02.112 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4e8a1e640 10 client.4472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:02.112 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4e8a1e640 10 monclient: _renew_subs 2024-03-29T11:31:02.112 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4e8a1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:02.112 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4e8a1e640 1 -- 172.21.15.135:0/1218828220 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e406eae0 con 0x7fd4e41457f0 2024-03-29T11:31:02.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4e8a1e640 1 librados: init done 2024-03-29T11:31:02.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.086+0000 7fd4e0d99640 4 set_mon_vals no callback set 2024-03-29T11:31:02.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.088+0000 7fd4e8a1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:02.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.088+0000 7fd4e8a1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:02.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.088+0000 7fd4e8a1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:02.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.088+0000 7fd4e8a1e640 1 -- 172.21.15.135:0/1218828220 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e4075f10 con 0x7fd4e41457f0 2024-03-29T11:31:02.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.089+0000 7fd4d2ffd640 1 -- 172.21.15.135:0/1218828220 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fd4d801ad70 con 0x7fd4e41457f0 2024-03-29T11:31:02.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.090+0000 7fd4d2ffd640 1 -- 172.21.15.135:0/1218828220 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7fd4d8084b40 con 0x7fd4e41457f0 2024-03-29T11:31:02.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.090+0000 7fd4d2ffd640 10 client.4472.objecter ms_dispatch 0x7fd4e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:31:02.114 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.090+0000 7fd4d2ffd640 3 client.4472.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:31:02.114 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.090+0000 7fd4e259c640 10 monclient: get_auth_request con 0x7fd4c406e000 auth_method 0 2024-03-29T11:31:02.114 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.090+0000 7fd4d2ffd640 3 client.4472.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:31:02.114 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.090+0000 7fd4d2ffd640 20 client.4472.objecter dump_active .. 0 homeless 2024-03-29T11:31:02.114 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.091+0000 7fd4d2ffd640 10 client.4472.objecter ms_handle_connect 0x7fd4c406e000 2024-03-29T11:31:02.114 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.097+0000 7fd4d2ffd640 1 -- 172.21.15.135:0/1218828220 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd4d8050420 con 0x7fd4e41457f0 2024-03-29T11:31:02.114 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.097+0000 7fd4d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:02.114 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.097+0000 7fd4d2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:02.479 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.486+0000 7fd4e8a1e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:02.479 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.486+0000 7fd4e8a1e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:02.479 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.486+0000 7fd4e8a1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:02.479 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.486+0000 7fd4e8a1e640 1 -- 172.21.15.135:0/1218828220 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4e41deb80 con 0x7fd4e41457f0 2024-03-29T11:31:02.480 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.487+0000 7fd4d2ffd640 1 -- 172.21.15.135:0/1218828220 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7fd4d8052350 con 0x7fd4e41457f0 2024-03-29T11:31:02.480 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.487+0000 7fd4d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:02.480 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.487+0000 7fd4d2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:02.480 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:31:02.480 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:02.483 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.490+0000 7fd4e8a1e640 10 librados: watch_flush enter 2024-03-29T11:31:02.484 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.490+0000 7fd4e8a1e640 10 librados: watch_flush exit 2024-03-29T11:31:02.484 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.490+0000 7fd4e8a1e640 20 client.4472.objecter shutdown clearing up homeless session... 2024-03-29T11:31:02.484 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.490+0000 7fd4e8a1e640 10 client.4472.objecter successfully canceled tick 2024-03-29T11:31:02.484 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.491+0000 7fd4e8a1e640 1 -- 172.21.15.135:0/1218828220 >> v1:172.21.15.135:6816/3469875285 conn(0x7fd4c406e000 legacy=0x7fd4c40704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:02.484 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.491+0000 7fd4e8a1e640 10 monclient: shutdown 2024-03-29T11:31:02.484 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.491+0000 7fd4e8a1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:02.484 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.491+0000 7fd4e8a1e640 1 -- 172.21.15.135:0/1218828220 >> v1:172.21.15.135:6790/0 conn(0x7fd4e41457f0 legacy=0x7fd4e41def60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:02.485 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.491+0000 7fd4e8a1e640 1 -- 172.21.15.135:0/1218828220 shutdown_connections 2024-03-29T11:31:02.485 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.491+0000 7fd4e8a1e640 1 -- 172.21.15.135:0/1218828220 >> 172.21.15.135:0/1218828220 conn(0x7fd4e4066a00 msgr2=0x7fd4e413cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:02.485 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.491+0000 7fd4e8a1e640 1 -- 172.21.15.135:0/1218828220 shutdown_connections 2024-03-29T11:31:02.485 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.491+0000 7fd4e8a1e640 1 -- 172.21.15.135:0/1218828220 wait complete. 2024-03-29T11:31:02.485 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:02.492+0000 7fd4e8a1e640 1 librados: shutdown 2024-03-29T11:31:02.500 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:09.307 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:09.438 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.445+0000 7f3f98893640 10 monclient: get_monmap_and_config 2024-03-29T11:31:09.439 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.445+0000 7f3f98893640 10 monclient: build_initial_monmap 2024-03-29T11:31:09.439 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.445+0000 7f3f98893640 10 monclient: monmap: 2024-03-29T11:31:09.439 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:31:09.439 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:09.439 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:31:09.446378+0000 2024-03-29T11:31:09.439 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:31:09.446378+0000 2024-03-29T11:31:09.439 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:09.439 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:09.439 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:31:09.439 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:31:09.439 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:31:09.440 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:09.440 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.446+0000 7f3f98893640 1 Processor -- start 2024-03-29T11:31:09.440 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.446+0000 7f3f98893640 1 -- start start 2024-03-29T11:31:09.440 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.446+0000 7f3f98893640 10 monclient: init 2024-03-29T11:31:09.441 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.447+0000 7f3f98893640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:09.441 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.447+0000 7f3f98893640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:31:09.441 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.447+0000 7f3f98893640 10 monclient(hunting): picked mon.noname-b con 0x7f3f941422b0 addr v1:172.21.15.135:6790/0 2024-03-29T11:31:09.441 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.447+0000 7f3f98893640 10 monclient(hunting): picked mon.noname-a con 0x7f3f94142690 addr v1:172.21.15.135:6789/0 2024-03-29T11:31:09.441 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.447+0000 7f3f98893640 10 monclient(hunting): picked mon.noname-c con 0x7f3f940b17d0 addr v1:172.21.15.194:6789/0 2024-03-29T11:31:09.442 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.447+0000 7f3f98893640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f94142c80 con 0x7f3f94142690 2024-03-29T11:31:09.442 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.447+0000 7f3f98893640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f940aae40 con 0x7f3f940b17d0 2024-03-29T11:31:09.442 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.447+0000 7f3f98893640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f9413fbd0 con 0x7f3f941422b0 2024-03-29T11:31:09.442 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.447+0000 7f3f98893640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:09.444 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.447+0000 7f3f98893640 10 monclient(hunting): authenticate will time out at 1176.816032s 2024-03-29T11:31:09.444 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.448+0000 7f3f9259c640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f3f941422b0 0x7f3f9413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:50892/0 (socket says 172.21.15.135:50892) 2024-03-29T11:31:09.444 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.448+0000 7f3f9259c640 1 -- 172.21.15.135:0/3056358859 learned_addr learned my addr 172.21.15.135:0/3056358859 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:31:09.444 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.448+0000 7f3f9159a640 1 -- 172.21.15.135:0/3056358859 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4281657267 0 0) 0x7f3f9413fbd0 con 0x7f3f941422b0 2024-03-29T11:31:09.444 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.449+0000 7f3f9159a640 10 monclient(hunting): _negotiate 2024-03-29T11:31:09.444 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.449+0000 7f3f9159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:09.444 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.449+0000 7f3f9159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:09.444 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.449+0000 7f3f9159a640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:31:09.445 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.449+0000 7f3f9159a640 1 -- 172.21.15.135:0/3056358859 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f7c003520 con 0x7f3f941422b0 2024-03-29T11:31:09.445 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.449+0000 7f3f9159a640 1 -- 172.21.15.135:0/3056358859 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1252707978 0 0) 0x7f3f940aae40 con 0x7f3f940b17d0 2024-03-29T11:31:09.445 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.449+0000 7f3f9159a640 10 monclient(hunting): _negotiate 2024-03-29T11:31:09.445 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.449+0000 7f3f9159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:09.445 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.449+0000 7f3f9159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:09.445 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.449+0000 7f3f9159a640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:31:09.445 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.449+0000 7f3f9159a640 1 -- 172.21.15.135:0/3056358859 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f9413fbd0 con 0x7f3f940b17d0 2024-03-29T11:31:09.445 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.449+0000 7f3f9159a640 1 -- 172.21.15.135:0/3056358859 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1852875310 0 0) 0x7f3f9413fbd0 con 0x7f3f940b17d0 2024-03-29T11:31:09.445 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.449+0000 7f3f9159a640 1 -- 172.21.15.135:0/3056358859 >> v1:172.21.15.135:6790/0 conn(0x7f3f941422b0 legacy=0x7f3f9413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:09.445 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.449+0000 7f3f9159a640 1 -- 172.21.15.135:0/3056358859 >> v1:172.21.15.135:6789/0 conn(0x7f3f94142690 legacy=0x7f3f940b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:31:09.445 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.450+0000 7f3f9159a640 10 monclient: _finish_hunting 0 2024-03-29T11:31:09.445 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.450+0000 7f3f9159a640 1 monclient: found mon.noname-c 2024-03-29T11:31:09.445 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.450+0000 7f3f9159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:09.446 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.450+0000 7f3f9159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:31:09.446 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.450+0000 7f3f9159a640 1 -- 172.21.15.135:0/3056358859 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f94151940 con 0x7f3f940b17d0 2024-03-29T11:31:09.446 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.450+0000 7f3f9159a640 10 monclient: _finish_auth 0 2024-03-29T11:31:09.446 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.450+0000 7f3f9159a640 10 monclient: _check_auth_tickets 2024-03-29T11:31:09.446 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.450+0000 7f3f9159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:09.446 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.450+0000 7f3f9159a640 1 -- 172.21.15.135:0/3056358859 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f3f8c001970 con 0x7f3f940b17d0 2024-03-29T11:31:09.446 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.450+0000 7f3f98893640 5 monclient: authenticate success, global_id 4477 2024-03-29T11:31:09.446 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.450+0000 7f3f9159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:09.446 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.450+0000 7f3f9159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:09.446 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.450+0000 7f3f9159a640 10 monclient: dump: 2024-03-29T11:31:09.446 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:09.446 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:09.446 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:09.446 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:09.447 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:09.447 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:09.447 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:09.447 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:09.447 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:09.447 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:09.447 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.451+0000 7f3f98893640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:09.449 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.452+0000 7f3f9159a640 1 -- 172.21.15.135:0/3056358859 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f8c001ad0 con 0x7f3f940b17d0 2024-03-29T11:31:09.449 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.452+0000 7f3f9159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:09.450 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.452+0000 7f3f9159a640 1 -- 172.21.15.135:0/3056358859 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f3f8c001ea0 con 0x7f3f940b17d0 2024-03-29T11:31:09.450 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.452+0000 7f3f9159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:09.450 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.452+0000 7f3f9159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:09.450 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.452+0000 7f3f9159a640 10 monclient: dump: 2024-03-29T11:31:09.450 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:09.450 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:09.450 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:09.450 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:09.450 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:09.450 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:09.450 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:09.450 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:09.450 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:09.450 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:09.451 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.452+0000 7f3f98893640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:09.451 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.452+0000 7f3f98893640 4 set_mon_vals no callback set 2024-03-29T11:31:09.451 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.453+0000 7f3f98893640 10 monclient: shutdown 2024-03-29T11:31:09.451 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.453+0000 7f3f98893640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:09.451 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.453+0000 7f3f98893640 1 -- 172.21.15.135:0/3056358859 >> v1:172.21.15.194:6789/0 conn(0x7f3f940b17d0 legacy=0x7f3f940b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:09.451 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.453+0000 7f3f98893640 1 -- 172.21.15.135:0/3056358859 shutdown_connections 2024-03-29T11:31:09.451 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.453+0000 7f3f98893640 1 -- 172.21.15.135:0/3056358859 >> 172.21.15.135:0/3056358859 conn(0x7f3f94066a00 msgr2=0x7f3f94066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:09.451 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.454+0000 7f3f98893640 1 -- 172.21.15.135:0/3056358859 shutdown_connections 2024-03-29T11:31:09.451 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.454+0000 7f3f98893640 1 -- 172.21.15.135:0/3056358859 wait complete. 2024-03-29T11:31:09.451 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.455+0000 7f3f98893640 10 monclient: build_initial_monmap 2024-03-29T11:31:09.451 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.455+0000 7f3f98893640 10 monclient: monmap: 2024-03-29T11:31:09.451 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:31:09.451 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:09.451 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:31:09.451 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:31:09.452 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:09.452 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:09.452 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:31:09.452 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:31:09.452 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:31:09.452 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:09.452 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.456+0000 7f3f98893640 1 librados: starting msgr at 2024-03-29T11:31:09.452 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.456+0000 7f3f98893640 1 librados: starting objecter 2024-03-29T11:31:09.452 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.456+0000 7f3f98893640 1 Processor -- start 2024-03-29T11:31:09.452 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.456+0000 7f3f98893640 1 -- start start 2024-03-29T11:31:09.452 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.456+0000 7f3f98893640 1 librados: setting wanted keys 2024-03-29T11:31:09.452 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.456+0000 7f3f98893640 1 librados: calling monclient init 2024-03-29T11:31:09.452 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.456+0000 7f3f98893640 10 monclient: init 2024-03-29T11:31:09.457 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.457+0000 7f3f98893640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:09.457 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.457+0000 7f3f98893640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:31:09.457 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.457+0000 7f3f98893640 10 monclient(hunting): picked mon.noname-c con 0x7f3f940b17d0 addr v1:172.21.15.135:6790/0 2024-03-29T11:31:09.457 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.457+0000 7f3f9259c640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f3f940b17d0 0x7f3f941ded30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:51652/0 (socket says 172.21.15.135:51652) 2024-03-29T11:31:09.457 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.457+0000 7f3f9259c640 1 -- 172.21.15.135:0/2906611724 learned_addr learned my addr 172.21.15.135:0/2906611724 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:31:09.457 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.457+0000 7f3f98893640 10 monclient(hunting): picked mon.noname-a con 0x7f3f941422b0 addr v1:172.21.15.135:6789/0 2024-03-29T11:31:09.457 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.457+0000 7f3f98893640 10 monclient(hunting): picked mon.noname-b con 0x7f3f94142690 addr v1:172.21.15.194:6789/0 2024-03-29T11:31:09.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.457+0000 7f3f98893640 1 -- 172.21.15.135:0/2906611724 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f9406e850 con 0x7f3f941422b0 2024-03-29T11:31:09.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.457+0000 7f3f98893640 1 -- 172.21.15.135:0/2906611724 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f9406eaa0 con 0x7f3f94142690 2024-03-29T11:31:09.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.457+0000 7f3f82ffd640 10 client.?.objecter ms_handle_connect 0x7f3f940b17d0 2024-03-29T11:31:09.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.457+0000 7f3f82ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:09.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.457+0000 7f3f98893640 1 -- 172.21.15.135:0/2906611724 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f9406ed80 con 0x7f3f940b17d0 2024-03-29T11:31:09.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.457+0000 7f3f98893640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:09.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.457+0000 7f3f98893640 10 monclient(hunting): authenticate will time out at 1176.825823s 2024-03-29T11:31:09.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.457+0000 7f3f82ffd640 1 -- 172.21.15.135:0/2906611724 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581854736 0 0) 0x7f3f9406ed80 con 0x7f3f940b17d0 2024-03-29T11:31:09.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.457+0000 7f3f82ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:09.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.457+0000 7f3f82ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:09.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.457+0000 7f3f82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:09.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.457+0000 7f3f82ffd640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:31:09.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.457+0000 7f3f82ffd640 1 -- 172.21.15.135:0/2906611724 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f74004eb0 con 0x7f3f940b17d0 2024-03-29T11:31:09.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.458+0000 7f3f82ffd640 1 -- 172.21.15.135:0/2906611724 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3850208445 0 0) 0x7f3f74004eb0 con 0x7f3f940b17d0 2024-03-29T11:31:09.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.458+0000 7f3f82ffd640 1 -- 172.21.15.135:0/2906611724 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f9406ed80 con 0x7f3f940b17d0 2024-03-29T11:31:09.459 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.458+0000 7f3f82ffd640 1 -- 172.21.15.135:0/2906611724 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f3f84001b50 con 0x7f3f940b17d0 2024-03-29T11:31:09.459 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.458+0000 7f3f82ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:09.459 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.458+0000 7f3f82ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:09.459 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.458+0000 7f3f82ffd640 10 monclient(hunting): dump: 2024-03-29T11:31:09.459 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:09.459 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:09.459 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:09.459 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:09.459 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:09.459 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:09.459 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:09.459 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:09.459 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:09.459 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:09.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.458+0000 7f3f82ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:09.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.458+0000 7f3f82ffd640 1 -- 172.21.15.135:0/2906611724 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3427984276 0 0) 0x7f3f9406ed80 con 0x7f3f940b17d0 2024-03-29T11:31:09.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.458+0000 7f3f82ffd640 1 -- 172.21.15.135:0/2906611724 >> v1:172.21.15.194:6789/0 conn(0x7f3f94142690 legacy=0x7f3f941e7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:09.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.458+0000 7f3f82ffd640 1 -- 172.21.15.135:0/2906611724 >> v1:172.21.15.135:6789/0 conn(0x7f3f941422b0 legacy=0x7f3f9412ec40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:09.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.461+0000 7f3f82ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:31:09.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.461+0000 7f3f82ffd640 1 monclient: found mon.c 2024-03-29T11:31:09.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.461+0000 7f3f82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:09.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.461+0000 7f3f82ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:09.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.461+0000 7f3f82ffd640 1 -- 172.21.15.135:0/2906611724 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f9406f060 con 0x7f3f940b17d0 2024-03-29T11:31:09.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.461+0000 7f3f82ffd640 10 monclient: _finish_auth 0 2024-03-29T11:31:09.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.461+0000 7f3f82ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:31:09.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.461+0000 7f3f82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:09.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.461+0000 7f3f98893640 5 monclient: authenticate success, global_id 4478 2024-03-29T11:31:09.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.461+0000 7f3f98893640 10 monclient: _renew_subs 2024-03-29T11:31:09.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.461+0000 7f3f98893640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:09.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.461+0000 7f3f98893640 1 -- 172.21.15.135:0/2906611724 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f940b7b50 con 0x7f3f940b17d0 2024-03-29T11:31:09.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.461+0000 7f3f98893640 10 client.4478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:09.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.461+0000 7f3f98893640 10 monclient: _renew_subs 2024-03-29T11:31:09.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.461+0000 7f3f98893640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:09.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.461+0000 7f3f98893640 1 -- 172.21.15.135:0/2906611724 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f94071bb0 con 0x7f3f940b17d0 2024-03-29T11:31:09.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.461+0000 7f3f98893640 1 librados: init done 2024-03-29T11:31:09.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.461+0000 7f3f82ffd640 1 -- 172.21.15.135:0/2906611724 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f840012d0 con 0x7f3f940b17d0 2024-03-29T11:31:09.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.462+0000 7f3f82ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:09.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.462+0000 7f3f82ffd640 1 -- 172.21.15.135:0/2906611724 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f3f84004c30 con 0x7f3f940b17d0 2024-03-29T11:31:09.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.462+0000 7f3f82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:09.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.462+0000 7f3f82ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:09.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.462+0000 7f3f82ffd640 10 monclient: dump: 2024-03-29T11:31:09.461 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:09.461 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:09.461 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:09.462 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:09.462 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:09.462 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:09.462 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:09.462 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:09.462 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:09.462 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:09.462 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.462+0000 7f3f90d99640 4 set_mon_vals no callback set 2024-03-29T11:31:09.462 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.463+0000 7f3f98893640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:09.462 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.463+0000 7f3f98893640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:09.462 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.463+0000 7f3f98893640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:09.462 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.463+0000 7f3f98893640 1 -- 172.21.15.135:0/2906611724 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f78005080 con 0x7f3f940b17d0 2024-03-29T11:31:09.462 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.464+0000 7f3f82ffd640 1 -- 172.21.15.135:0/2906611724 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f3f84004d90 con 0x7f3f940b17d0 2024-03-29T11:31:09.463 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.465+0000 7f3f82ffd640 1 -- 172.21.15.135:0/2906611724 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f3f84084c60 con 0x7f3f940b17d0 2024-03-29T11:31:09.463 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.465+0000 7f3f91d9b640 10 monclient: get_auth_request con 0x7f3f7406dc10 auth_method 0 2024-03-29T11:31:09.463 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.465+0000 7f3f82ffd640 10 client.4478.objecter ms_dispatch 0x7f3f94000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:31:09.463 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.465+0000 7f3f82ffd640 3 client.4478.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:31:09.463 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.465+0000 7f3f82ffd640 3 client.4478.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:31:09.463 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.465+0000 7f3f82ffd640 20 client.4478.objecter dump_active .. 0 homeless 2024-03-29T11:31:09.463 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.465+0000 7f3f82ffd640 10 client.4478.objecter ms_handle_connect 0x7f3f7406dc10 2024-03-29T11:31:09.465 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.472+0000 7f3f82ffd640 1 -- 172.21.15.135:0/2906611724 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3f84050540 con 0x7f3f940b17d0 2024-03-29T11:31:09.465 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.472+0000 7f3f82ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:09.465 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.472+0000 7f3f82ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:09.780 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.787+0000 7f3f98893640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:09.780 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.787+0000 7f3f98893640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:09.780 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.787+0000 7f3f98893640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:09.781 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.787+0000 7f3f98893640 1 -- 172.21.15.135:0/2906611724 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f78005330 con 0x7f3f940b17d0 2024-03-29T11:31:09.781 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.788+0000 7f3f82ffd640 1 -- 172.21.15.135:0/2906611724 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f3f84052470 con 0x7f3f940b17d0 2024-03-29T11:31:09.781 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.788+0000 7f3f82ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:09.781 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.788+0000 7f3f82ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:09.781 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:31:09.782 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:09.785 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.791+0000 7f3f98893640 10 librados: watch_flush enter 2024-03-29T11:31:09.785 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.791+0000 7f3f98893640 10 librados: watch_flush exit 2024-03-29T11:31:09.785 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.791+0000 7f3f98893640 20 client.4478.objecter shutdown clearing up homeless session... 2024-03-29T11:31:09.785 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.791+0000 7f3f98893640 10 client.4478.objecter successfully canceled tick 2024-03-29T11:31:09.785 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.791+0000 7f3f98893640 1 -- 172.21.15.135:0/2906611724 >> v1:172.21.15.135:6816/3469875285 conn(0x7f3f7406dc10 legacy=0x7f3f740700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:09.785 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.791+0000 7f3f98893640 10 monclient: shutdown 2024-03-29T11:31:09.785 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.791+0000 7f3f98893640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:09.785 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.791+0000 7f3f98893640 1 -- 172.21.15.135:0/2906611724 >> v1:172.21.15.135:6790/0 conn(0x7f3f940b17d0 legacy=0x7f3f941ded30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:09.786 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.791+0000 7f3f98893640 1 -- 172.21.15.135:0/2906611724 shutdown_connections 2024-03-29T11:31:09.786 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.791+0000 7f3f98893640 1 -- 172.21.15.135:0/2906611724 >> 172.21.15.135:0/2906611724 conn(0x7f3f94066a00 msgr2=0x7f3f940b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:09.786 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.792+0000 7f3f98893640 1 -- 172.21.15.135:0/2906611724 shutdown_connections 2024-03-29T11:31:09.786 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.792+0000 7f3f98893640 1 -- 172.21.15.135:0/2906611724 wait complete. 2024-03-29T11:31:09.786 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:09.792+0000 7f3f98893640 1 librados: shutdown 2024-03-29T11:31:09.798 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:16.605 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:16.751 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.756+0000 7fc4cfbed640 10 monclient: get_monmap_and_config 2024-03-29T11:31:16.751 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.756+0000 7fc4cfbed640 10 monclient: build_initial_monmap 2024-03-29T11:31:16.751 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.756+0000 7fc4cfbed640 10 monclient: monmap: 2024-03-29T11:31:16.751 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:31:16.751 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:16.751 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:31:16.758121+0000 2024-03-29T11:31:16.751 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:31:16.758121+0000 2024-03-29T11:31:16.751 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:16.751 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:16.752 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:31:16.752 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:31:16.752 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:31:16.752 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:16.752 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.757+0000 7fc4cfbed640 1 Processor -- start 2024-03-29T11:31:16.756 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.758+0000 7fc4cfbed640 1 -- start start 2024-03-29T11:31:16.756 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.758+0000 7fc4cfbed640 10 monclient: init 2024-03-29T11:31:16.756 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.758+0000 7fc4cfbed640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:16.757 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.758+0000 7fc4cfbed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:31:16.757 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.758+0000 7fc4cfbed640 10 monclient(hunting): picked mon.noname-a con 0x7fc4c80b0770 addr v1:172.21.15.135:6789/0 2024-03-29T11:31:16.757 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.758+0000 7fc4cfbed640 10 monclient(hunting): picked mon.noname-b con 0x7fc4c814a700 addr v1:172.21.15.135:6790/0 2024-03-29T11:31:16.757 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.759+0000 7fc4cfbed640 10 monclient(hunting): picked mon.noname-c con 0x7fc4c80ac5a0 addr v1:172.21.15.194:6789/0 2024-03-29T11:31:16.757 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.759+0000 7fc4cfbed640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c80aba70 con 0x7fc4c80b0770 2024-03-29T11:31:16.757 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.759+0000 7fc4cfbed640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c80aecb0 con 0x7fc4c80ac5a0 2024-03-29T11:31:16.757 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.759+0000 7fc4cfbed640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c80af150 con 0x7fc4c814a700 2024-03-29T11:31:16.757 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.759+0000 7fc4cfbed640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:16.757 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.759+0000 7fc4cfbed640 10 monclient(hunting): authenticate will time out at 1184.128083s 2024-03-29T11:31:16.757 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.760+0000 7fc4cd188640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7fc4c814a700 0x7fc4c80abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:54176/0 (socket says 172.21.15.135:54176) 2024-03-29T11:31:16.757 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.760+0000 7fc4cd188640 1 -- 172.21.15.135:0/4074973988 learned_addr learned my addr 172.21.15.135:0/4074973988 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:31:16.757 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.761+0000 7fc4cc987640 1 -- 172.21.15.135:0/4074973988 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181828016 0 0) 0x7fc4c80af150 con 0x7fc4c814a700 2024-03-29T11:31:16.757 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.762+0000 7fc4cc987640 10 monclient(hunting): _negotiate 2024-03-29T11:31:16.758 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.762+0000 7fc4cc987640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:16.758 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.762+0000 7fc4cc987640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:16.758 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.762+0000 7fc4cc987640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:31:16.758 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.762+0000 7fc4cc987640 1 -- 172.21.15.135:0/4074973988 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4b4003520 con 0x7fc4c814a700 2024-03-29T11:31:16.759 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.765+0000 7fc4cc987640 1 -- 172.21.15.135:0/4074973988 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3250615676 0 0) 0x7fc4b4003520 con 0x7fc4c814a700 2024-03-29T11:31:16.759 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.765+0000 7fc4cc987640 1 -- 172.21.15.135:0/4074973988 >> v1:172.21.15.194:6789/0 conn(0x7fc4c80ac5a0 legacy=0x7fc4c8151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:16.760 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.765+0000 7fc4cc987640 1 -- 172.21.15.135:0/4074973988 >> v1:172.21.15.135:6789/0 conn(0x7fc4c80b0770 legacy=0x7fc4c8149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:16.760 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.765+0000 7fc4cc987640 10 monclient: _finish_hunting 0 2024-03-29T11:31:16.760 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.765+0000 7fc4cc987640 1 monclient: found mon.noname-b 2024-03-29T11:31:16.760 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.765+0000 7fc4cc987640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:16.760 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.765+0000 7fc4cc987640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:31:16.760 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.765+0000 7fc4cc987640 1 -- 172.21.15.135:0/4074973988 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c80ac9d0 con 0x7fc4c814a700 2024-03-29T11:31:16.760 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.765+0000 7fc4cc987640 10 monclient: _finish_auth 0 2024-03-29T11:31:16.760 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.765+0000 7fc4cc987640 10 monclient: _check_auth_tickets 2024-03-29T11:31:16.760 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.765+0000 7fc4cc987640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:16.760 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.765+0000 7fc4cc987640 1 -- 172.21.15.135:0/4074973988 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fc4b8001690 con 0x7fc4c814a700 2024-03-29T11:31:16.760 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.765+0000 7fc4cc987640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:16.760 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.765+0000 7fc4cc987640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:16.760 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.765+0000 7fc4cc987640 10 monclient: dump: 2024-03-29T11:31:16.760 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:16.761 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:16.761 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:16.761 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:16.761 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:16.761 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:16.761 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:16.761 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:16.761 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:16.761 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:16.761 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.765+0000 7fc4cfbed640 5 monclient: authenticate success, global_id 4481 2024-03-29T11:31:16.761 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.765+0000 7fc4cfbed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:16.762 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.766+0000 7fc4cc987640 1 -- 172.21.15.135:0/4074973988 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4b8001860 con 0x7fc4c814a700 2024-03-29T11:31:16.762 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.766+0000 7fc4cc987640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:16.762 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.766+0000 7fc4cc987640 1 -- 172.21.15.135:0/4074973988 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fc4b8001cb0 con 0x7fc4c814a700 2024-03-29T11:31:16.762 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.766+0000 7fc4cfbed640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:16.763 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.766+0000 7fc4cfbed640 4 set_mon_vals no callback set 2024-03-29T11:31:16.763 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.766+0000 7fc4cc987640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:16.763 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.766+0000 7fc4cc987640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:16.763 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.766+0000 7fc4cc987640 10 monclient: dump: 2024-03-29T11:31:16.763 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:16.763 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:16.763 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:16.763 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:16.763 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:16.763 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:16.763 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:16.763 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:16.763 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:16.763 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:16.763 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.767+0000 7fc4cfbed640 10 monclient: shutdown 2024-03-29T11:31:16.764 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.767+0000 7fc4cfbed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:16.764 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.767+0000 7fc4cfbed640 1 -- 172.21.15.135:0/4074973988 >> v1:172.21.15.135:6790/0 conn(0x7fc4c814a700 legacy=0x7fc4c80abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:16.764 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.769+0000 7fc4cfbed640 1 -- 172.21.15.135:0/4074973988 shutdown_connections 2024-03-29T11:31:16.764 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.769+0000 7fc4cfbed640 1 -- 172.21.15.135:0/4074973988 >> 172.21.15.135:0/4074973988 conn(0x7fc4c806b660 msgr2=0x7fc4c806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:16.765 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.770+0000 7fc4cfbed640 1 -- 172.21.15.135:0/4074973988 shutdown_connections 2024-03-29T11:31:16.765 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.770+0000 7fc4cfbed640 1 -- 172.21.15.135:0/4074973988 wait complete. 2024-03-29T11:31:16.767 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.772+0000 7fc4cfbed640 10 monclient: build_initial_monmap 2024-03-29T11:31:16.767 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.772+0000 7fc4cfbed640 10 monclient: monmap: 2024-03-29T11:31:16.767 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:31:16.767 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:16.767 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:31:16.768 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:31:16.768 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:16.768 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:16.768 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:31:16.768 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:31:16.768 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:31:16.768 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:16.768 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.772+0000 7fc4cfbed640 1 librados: starting msgr at 2024-03-29T11:31:16.768 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.772+0000 7fc4cfbed640 1 librados: starting objecter 2024-03-29T11:31:16.768 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.773+0000 7fc4cfbed640 1 Processor -- start 2024-03-29T11:31:16.770 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.774+0000 7fc4cfbed640 1 -- start start 2024-03-29T11:31:16.770 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.774+0000 7fc4cfbed640 1 librados: setting wanted keys 2024-03-29T11:31:16.770 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.774+0000 7fc4cfbed640 1 librados: calling monclient init 2024-03-29T11:31:16.770 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.774+0000 7fc4cfbed640 10 monclient: init 2024-03-29T11:31:16.770 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.775+0000 7fc4cfbed640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:16.770 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.775+0000 7fc4cfbed640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:31:16.770 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.775+0000 7fc4cfbed640 10 monclient(hunting): picked mon.noname-c con 0x7fc4c80ac5a0 addr v1:172.21.15.135:6790/0 2024-03-29T11:31:16.770 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.775+0000 7fc4cfbed640 10 monclient(hunting): picked mon.noname-b con 0x7fc4c80b0770 addr v1:172.21.15.194:6789/0 2024-03-29T11:31:16.770 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.775+0000 7fc4cfbed640 10 monclient(hunting): picked mon.noname-a con 0x7fc4c814a700 addr v1:172.21.15.135:6789/0 2024-03-29T11:31:16.770 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.775+0000 7fc4cfbed640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c812a230 con 0x7fc4c814a700 2024-03-29T11:31:16.770 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.775+0000 7fc4cfbed640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c81deb10 con 0x7fc4c80b0770 2024-03-29T11:31:16.771 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.775+0000 7fc4cfbed640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c8125550 con 0x7fc4c80ac5a0 2024-03-29T11:31:16.771 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.775+0000 7fc4cfbed640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:16.771 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.775+0000 7fc4cfbed640 10 monclient(hunting): authenticate will time out at 1184.144222s 2024-03-29T11:31:16.772 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.778+0000 7fc4cd188640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fc4c80b0770 0x7fc4c8147040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:50795/0 (socket says 172.21.15.135:50795) 2024-03-29T11:31:16.773 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.778+0000 7fc4cd188640 1 -- 172.21.15.135:0/202458080 learned_addr learned my addr 172.21.15.135:0/202458080 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:31:16.773 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.778+0000 7fc4be7fc640 10 client.?.objecter ms_handle_connect 0x7fc4c80b0770 2024-03-29T11:31:16.773 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.778+0000 7fc4be7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:16.774 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.779+0000 7fc4be7fc640 10 client.?.objecter ms_handle_connect 0x7fc4c80ac5a0 2024-03-29T11:31:16.774 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.779+0000 7fc4be7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:16.774 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.779+0000 7fc4be7fc640 1 -- 172.21.15.135:0/202458080 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1347753229 0 0) 0x7fc4c81deb10 con 0x7fc4c80b0770 2024-03-29T11:31:16.774 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.779+0000 7fc4be7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:31:16.774 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.779+0000 7fc4be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:16.774 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.779+0000 7fc4be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:16.775 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.779+0000 7fc4be7fc640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:31:16.775 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.779+0000 7fc4be7fc640 1 -- 172.21.15.135:0/202458080 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4ac004eb0 con 0x7fc4c80b0770 2024-03-29T11:31:16.775 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.780+0000 7fc4be7fc640 10 client.?.objecter ms_handle_connect 0x7fc4c814a700 2024-03-29T11:31:16.775 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.780+0000 7fc4be7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:16.775 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.780+0000 7fc4be7fc640 1 -- 172.21.15.135:0/202458080 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941455195 0 0) 0x7fc4c8125550 con 0x7fc4c80ac5a0 2024-03-29T11:31:16.775 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.780+0000 7fc4be7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:31:16.780 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.780+0000 7fc4be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:16.780 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.780+0000 7fc4be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:16.780 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.780+0000 7fc4be7fc640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:31:16.780 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.780+0000 7fc4be7fc640 1 -- 172.21.15.135:0/202458080 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4c81deb10 con 0x7fc4c80ac5a0 2024-03-29T11:31:16.780 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.781+0000 7fc4be7fc640 1 -- 172.21.15.135:0/202458080 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3865663102 0 0) 0x7fc4c812a230 con 0x7fc4c814a700 2024-03-29T11:31:16.780 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.781+0000 7fc4be7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:31:16.780 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.781+0000 7fc4be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:16.781 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.781+0000 7fc4be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:16.781 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.781+0000 7fc4be7fc640 10 monclient(hunting): my global_id is 4464 2024-03-29T11:31:16.781 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.781+0000 7fc4be7fc640 1 -- 172.21.15.135:0/202458080 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4c8125550 con 0x7fc4c814a700 2024-03-29T11:31:16.781 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.781+0000 7fc4be7fc640 1 -- 172.21.15.135:0/202458080 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3102520304 0 0) 0x7fc4ac004eb0 con 0x7fc4c80b0770 2024-03-29T11:31:16.781 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.781+0000 7fc4be7fc640 1 -- 172.21.15.135:0/202458080 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4c812a230 con 0x7fc4c80b0770 2024-03-29T11:31:16.781 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.781+0000 7fc4be7fc640 1 -- 172.21.15.135:0/202458080 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 989078886 0 0) 0x7fc4c81deb10 con 0x7fc4c80ac5a0 2024-03-29T11:31:16.781 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.781+0000 7fc4be7fc640 1 -- 172.21.15.135:0/202458080 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4ac004eb0 con 0x7fc4c80ac5a0 2024-03-29T11:31:16.781 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.781+0000 7fc4be7fc640 1 -- 172.21.15.135:0/202458080 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fc4b8001b70 con 0x7fc4c80b0770 2024-03-29T11:31:16.781 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.781+0000 7fc4be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:16.781 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.782+0000 7fc4be7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:16.781 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.782+0000 7fc4be7fc640 10 monclient(hunting): dump: 2024-03-29T11:31:16.781 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:16.781 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:16.781 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:16.782 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:16.782 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:16.782 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:16.782 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:16.782 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:16.782 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:16.782 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:16.782 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.782+0000 7fc4be7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:16.782 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.782+0000 7fc4be7fc640 1 -- 172.21.15.135:0/202458080 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fc4c0001810 con 0x7fc4c80ac5a0 2024-03-29T11:31:16.782 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.782+0000 7fc4be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:16.782 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.782+0000 7fc4be7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:16.782 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.782+0000 7fc4be7fc640 10 monclient(hunting): dump: 2024-03-29T11:31:16.782 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:16.783 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:16.783 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:16.783 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:16.783 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:16.783 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:16.783 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:16.783 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:16.783 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:16.783 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:16.783 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.782+0000 7fc4be7fc640 1 -- 172.21.15.135:0/202458080 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 360073021 0 0) 0x7fc4c812a230 con 0x7fc4c80b0770 2024-03-29T11:31:16.783 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.782+0000 7fc4be7fc640 1 -- 172.21.15.135:0/202458080 >> v1:172.21.15.135:6790/0 conn(0x7fc4c80ac5a0 legacy=0x7fc4c81df030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:16.783 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.782+0000 7fc4be7fc640 1 -- 172.21.15.135:0/202458080 >> v1:172.21.15.135:6789/0 conn(0x7fc4c814a700 legacy=0x7fc4c81e6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:16.783 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.782+0000 7fc4be7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:31:16.783 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.782+0000 7fc4be7fc640 1 monclient: found mon.b 2024-03-29T11:31:16.784 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.782+0000 7fc4be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:16.784 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.782+0000 7fc4be7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:31:16.784 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.782+0000 7fc4be7fc640 1 -- 172.21.15.135:0/202458080 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c81286b0 con 0x7fc4c80b0770 2024-03-29T11:31:16.784 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.782+0000 7fc4be7fc640 10 monclient: _finish_auth 0 2024-03-29T11:31:16.784 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.782+0000 7fc4be7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:31:16.784 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.783+0000 7fc4be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:16.784 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.783+0000 7fc4cfbed640 5 monclient: authenticate success, global_id 4483 2024-03-29T11:31:16.784 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.783+0000 7fc4cfbed640 10 monclient: _renew_subs 2024-03-29T11:31:16.784 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.783+0000 7fc4cfbed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:31:16.784 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.783+0000 7fc4cfbed640 1 -- 172.21.15.135:0/202458080 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4c81df740 con 0x7fc4c80b0770 2024-03-29T11:31:16.784 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.783+0000 7fc4cfbed640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:16.784 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.783+0000 7fc4cfbed640 10 monclient: _renew_subs 2024-03-29T11:31:16.784 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.783+0000 7fc4cfbed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:31:16.784 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.783+0000 7fc4cfbed640 1 -- 172.21.15.135:0/202458080 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c8147930 con 0x7fc4c80b0770 2024-03-29T11:31:16.785 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.783+0000 7fc4cfbed640 1 librados: init done 2024-03-29T11:31:16.785 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.783+0000 7fc4be7fc640 1 -- 172.21.15.135:0/202458080 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4b8004a90 con 0x7fc4c80b0770 2024-03-29T11:31:16.785 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.783+0000 7fc4be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:16.785 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.783+0000 7fc4be7fc640 1 -- 172.21.15.135:0/202458080 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fc4b8001cd0 con 0x7fc4c80b0770 2024-03-29T11:31:16.785 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.783+0000 7fc4be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:16.785 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.783+0000 7fc4be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:16.785 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.783+0000 7fc4be7fc640 10 monclient: dump: 2024-03-29T11:31:16.785 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:16.785 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:16.785 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:16.785 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:16.785 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:16.785 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:16.785 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:16.785 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:16.785 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:16.786 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:16.786 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.784+0000 7fc4cfbed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:16.786 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.784+0000 7fc4cfbed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:16.786 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.784+0000 7fc4cfbed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:31:16.786 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.784+0000 7fc4cfbed640 1 -- 172.21.15.135:0/202458080 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c81288a0 con 0x7fc4c80b0770 2024-03-29T11:31:16.786 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.785+0000 7fc4bffff640 4 set_mon_vals no callback set 2024-03-29T11:31:16.786 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.789+0000 7fc4be7fc640 1 -- 172.21.15.135:0/202458080 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fc4b801ac10 con 0x7fc4c80b0770 2024-03-29T11:31:16.787 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.790+0000 7fc4be7fc640 1 -- 172.21.15.135:0/202458080 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7fc4b8084a90 con 0x7fc4c80b0770 2024-03-29T11:31:16.787 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.790+0000 7fc4be7fc640 10 client.4483.objecter ms_dispatch 0x7fc4c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:31:16.787 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.790+0000 7fc4be7fc640 3 client.4483.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:31:16.787 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.790+0000 7fc4be7fc640 3 client.4483.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:31:16.787 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.790+0000 7fc4cd989640 10 monclient: get_auth_request con 0x7fc4ac06ded0 auth_method 0 2024-03-29T11:31:16.787 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.790+0000 7fc4be7fc640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-29T11:31:16.787 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.791+0000 7fc4be7fc640 10 client.4483.objecter ms_handle_connect 0x7fc4ac06ded0 2024-03-29T11:31:16.788 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.793+0000 7fc4be7fc640 1 -- 172.21.15.135:0/202458080 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc4b8050370 con 0x7fc4c80b0770 2024-03-29T11:31:16.788 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.794+0000 7fc4be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:16.788 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:16.794+0000 7fc4be7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:17.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:17.117+0000 7fc4cfbed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:17.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:17.118+0000 7fc4cfbed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:17.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:17.118+0000 7fc4cfbed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:31:17.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:17.118+0000 7fc4cfbed640 1 -- 172.21.15.135:0/202458080 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4c80b0b50 con 0x7fc4c80b0770 2024-03-29T11:31:17.114 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:31:17.115 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:17.115 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:17.120+0000 7fc4be7fc640 1 -- 172.21.15.135:0/202458080 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7fc4b80522a0 con 0x7fc4c80b0770 2024-03-29T11:31:17.115 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:17.120+0000 7fc4be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:17.115 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:17.120+0000 7fc4be7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:17.118 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:17.123+0000 7fc4cfbed640 10 librados: watch_flush enter 2024-03-29T11:31:17.118 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:17.123+0000 7fc4cfbed640 10 librados: watch_flush exit 2024-03-29T11:31:17.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:17.123+0000 7fc4cfbed640 20 client.4483.objecter shutdown clearing up homeless session... 2024-03-29T11:31:17.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:17.123+0000 7fc4cfbed640 10 client.4483.objecter successfully canceled tick 2024-03-29T11:31:17.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:17.123+0000 7fc4cfbed640 1 -- 172.21.15.135:0/202458080 >> v1:172.21.15.135:6816/3469875285 conn(0x7fc4ac06ded0 legacy=0x7fc4ac070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:17.124+0000 7fc4cfbed640 10 monclient: shutdown 2024-03-29T11:31:17.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:17.124+0000 7fc4cfbed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:17.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:17.124+0000 7fc4cfbed640 1 -- 172.21.15.135:0/202458080 >> v1:172.21.15.194:6789/0 conn(0x7fc4c80b0770 legacy=0x7fc4c8147040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:17.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:17.124+0000 7fc4cfbed640 1 -- 172.21.15.135:0/202458080 shutdown_connections 2024-03-29T11:31:17.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:17.124+0000 7fc4cfbed640 1 -- 172.21.15.135:0/202458080 >> 172.21.15.135:0/202458080 conn(0x7fc4c806b660 msgr2=0x7fc4c80abe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:17.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:17.124+0000 7fc4cfbed640 1 -- 172.21.15.135:0/202458080 shutdown_connections 2024-03-29T11:31:17.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:17.124+0000 7fc4cfbed640 1 -- 172.21.15.135:0/202458080 wait complete. 2024-03-29T11:31:17.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:17.124+0000 7fc4cfbed640 1 librados: shutdown 2024-03-29T11:31:17.136 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:23.943 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:24.090 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.095+0000 7fad208ac640 10 monclient: get_monmap_and_config 2024-03-29T11:31:24.090 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.095+0000 7fad208ac640 10 monclient: build_initial_monmap 2024-03-29T11:31:24.090 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.095+0000 7fad208ac640 10 monclient: monmap: 2024-03-29T11:31:24.090 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:31:24.090 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:24.091 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:31:24.097498+0000 2024-03-29T11:31:24.091 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:31:24.097498+0000 2024-03-29T11:31:24.091 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:24.091 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:24.091 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:31:24.091 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:31:24.091 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:31:24.091 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:24.092 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.097+0000 7fad208ac640 1 Processor -- start 2024-03-29T11:31:24.092 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.097+0000 7fad208ac640 1 -- start start 2024-03-29T11:31:24.092 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.097+0000 7fad208ac640 10 monclient: init 2024-03-29T11:31:24.096 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.098+0000 7fad208ac640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:24.096 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.098+0000 7fad208ac640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:31:24.096 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.098+0000 7fad208ac640 10 monclient(hunting): picked mon.noname-c con 0x7fad14036a60 addr v1:172.21.15.194:6789/0 2024-03-29T11:31:24.096 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.098+0000 7fad208ac640 10 monclient(hunting): picked mon.noname-a con 0x7fad140c5de0 addr v1:172.21.15.135:6789/0 2024-03-29T11:31:24.096 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.098+0000 7fad208ac640 10 monclient(hunting): picked mon.noname-b con 0x7fad140c3070 addr v1:172.21.15.135:6790/0 2024-03-29T11:31:24.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.098+0000 7fad208ac640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad1402cc60 con 0x7fad140c5de0 2024-03-29T11:31:24.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.098+0000 7fad208ac640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad1402e040 con 0x7fad14036a60 2024-03-29T11:31:24.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.098+0000 7fad208ac640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad1402e320 con 0x7fad140c3070 2024-03-29T11:31:24.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.098+0000 7fad208ac640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:24.097 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.098+0000 7fad208ac640 10 monclient(hunting): authenticate will time out at 1191.467897s 2024-03-29T11:31:24.098 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.102+0000 7fad1bfff640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7fad140c3070 0x7fad140d0fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:51471/0 (socket says 172.21.15.135:51471) 2024-03-29T11:31:24.098 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.102+0000 7fad1bfff640 1 -- 172.21.15.135:0/2181431968 learned_addr learned my addr 172.21.15.135:0/2181431968 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:31:24.098 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.104+0000 7fad1a7fc640 1 -- 172.21.15.135:0/2181431968 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3915908203 0 0) 0x7fad1402e320 con 0x7fad140c3070 2024-03-29T11:31:24.098 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.104+0000 7fad1a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:31:24.098 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.104+0000 7fad1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:24.098 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.104+0000 7fad1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:24.099 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.104+0000 7fad1a7fc640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:31:24.099 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.104+0000 7fad1a7fc640 1 -- 172.21.15.135:0/2181431968 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad0c003520 con 0x7fad140c3070 2024-03-29T11:31:24.099 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.104+0000 7fad1a7fc640 1 -- 172.21.15.135:0/2181431968 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3930998455 0 0) 0x7fad1402cc60 con 0x7fad140c5de0 2024-03-29T11:31:24.099 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.104+0000 7fad1a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:31:24.099 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.104+0000 7fad1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:24.099 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.104+0000 7fad1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:24.099 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.104+0000 7fad1a7fc640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:31:24.099 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.104+0000 7fad1a7fc640 1 -- 172.21.15.135:0/2181431968 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad1402e320 con 0x7fad140c5de0 2024-03-29T11:31:24.099 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.104+0000 7fad1a7fc640 1 -- 172.21.15.135:0/2181431968 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1003772801 0 0) 0x7fad0c003520 con 0x7fad140c3070 2024-03-29T11:31:24.099 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.104+0000 7fad1a7fc640 1 -- 172.21.15.135:0/2181431968 >> v1:172.21.15.194:6789/0 conn(0x7fad14036a60 legacy=0x7fad140c56d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:31:24.101 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.105+0000 7fad1a7fc640 1 -- 172.21.15.135:0/2181431968 >> v1:172.21.15.135:6789/0 conn(0x7fad140c5de0 legacy=0x7fad140c2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:24.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.105+0000 7fad1a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:31:24.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.105+0000 7fad1a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:31:24.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.105+0000 7fad1a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:24.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.105+0000 7fad1a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:31:24.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.105+0000 7fad1a7fc640 1 -- 172.21.15.135:0/2181431968 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1402c830 con 0x7fad140c3070 2024-03-29T11:31:24.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.105+0000 7fad1a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:31:24.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.105+0000 7fad1a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:31:24.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.105+0000 7fad1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:24.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.105+0000 7fad1a7fc640 1 -- 172.21.15.135:0/2181431968 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fad080016e0 con 0x7fad140c3070 2024-03-29T11:31:24.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.105+0000 7fad1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:24.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.105+0000 7fad1a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:24.102 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.105+0000 7fad1a7fc640 10 monclient: dump: 2024-03-29T11:31:24.102 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:24.103 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:24.103 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:24.103 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:24.103 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:24.103 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:24.103 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:24.103 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:24.103 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:24.103 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:24.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.106+0000 7fad208ac640 5 monclient: authenticate success, global_id 4487 2024-03-29T11:31:24.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.106+0000 7fad208ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:24.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.107+0000 7fad1a7fc640 1 -- 172.21.15.135:0/2181431968 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad080018b0 con 0x7fad140c3070 2024-03-29T11:31:24.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.107+0000 7fad1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:24.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.107+0000 7fad1a7fc640 1 -- 172.21.15.135:0/2181431968 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fad08001cc0 con 0x7fad140c3070 2024-03-29T11:31:24.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.107+0000 7fad1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:24.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.107+0000 7fad1a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:24.104 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.107+0000 7fad1a7fc640 10 monclient: dump: 2024-03-29T11:31:24.104 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:24.104 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:24.104 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:24.104 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:24.104 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:24.105 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:24.105 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:24.105 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:24.105 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:24.105 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:24.105 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.107+0000 7fad208ac640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:24.105 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.107+0000 7fad208ac640 4 set_mon_vals no callback set 2024-03-29T11:31:24.107 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.108+0000 7fad208ac640 10 monclient: shutdown 2024-03-29T11:31:24.108 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.108+0000 7fad208ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:24.108 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.108+0000 7fad208ac640 1 -- 172.21.15.135:0/2181431968 >> v1:172.21.15.135:6790/0 conn(0x7fad140c3070 legacy=0x7fad140d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:24.108 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.108+0000 7fad208ac640 1 -- 172.21.15.135:0/2181431968 shutdown_connections 2024-03-29T11:31:24.108 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.108+0000 7fad208ac640 1 -- 172.21.15.135:0/2181431968 >> 172.21.15.135:0/2181431968 conn(0x7fad14005c00 msgr2=0x7fad14006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:24.108 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.109+0000 7fad208ac640 1 -- 172.21.15.135:0/2181431968 shutdown_connections 2024-03-29T11:31:24.108 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.109+0000 7fad208ac640 1 -- 172.21.15.135:0/2181431968 wait complete. 2024-03-29T11:31:24.108 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.110+0000 7fad208ac640 10 monclient: build_initial_monmap 2024-03-29T11:31:24.108 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.110+0000 7fad208ac640 10 monclient: monmap: 2024-03-29T11:31:24.108 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:31:24.108 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:24.108 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:31:24.108 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:31:24.108 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:24.108 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:24.109 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:31:24.109 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:31:24.109 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:31:24.109 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:24.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.110+0000 7fad208ac640 1 librados: starting msgr at 2024-03-29T11:31:24.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.110+0000 7fad208ac640 1 librados: starting objecter 2024-03-29T11:31:24.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.111+0000 7fad208ac640 1 Processor -- start 2024-03-29T11:31:24.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.111+0000 7fad208ac640 1 -- start start 2024-03-29T11:31:24.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.111+0000 7fad208ac640 1 librados: setting wanted keys 2024-03-29T11:31:24.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.111+0000 7fad208ac640 1 librados: calling monclient init 2024-03-29T11:31:24.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.111+0000 7fad208ac640 10 monclient: init 2024-03-29T11:31:24.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.112+0000 7fad208ac640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:24.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.112+0000 7fad208ac640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:31:24.109 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.112+0000 7fad208ac640 10 monclient(hunting): picked mon.noname-a con 0x7fad14036a60 addr v1:172.21.15.135:6789/0 2024-03-29T11:31:24.110 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.112+0000 7fad208ac640 10 monclient(hunting): picked mon.noname-b con 0x7fad140c3070 addr v1:172.21.15.194:6789/0 2024-03-29T11:31:24.110 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.112+0000 7fad208ac640 10 monclient(hunting): picked mon.noname-c con 0x7fad14007cd0 addr v1:172.21.15.135:6790/0 2024-03-29T11:31:24.110 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.112+0000 7fad208ac640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad14160e60 con 0x7fad14036a60 2024-03-29T11:31:24.110 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.112+0000 7fad208ac640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad14161d30 con 0x7fad140c3070 2024-03-29T11:31:24.110 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.112+0000 7fad208ac640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad14161100 con 0x7fad14007cd0 2024-03-29T11:31:24.110 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.112+0000 7fad208ac640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:24.110 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.112+0000 7fad208ac640 10 monclient(hunting): authenticate will time out at 1191.481217s 2024-03-29T11:31:24.112 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.116+0000 7fad1bfff640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7fad14007cd0 0x7fad1416b8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:51573/0 (socket says 172.21.15.135:51573) 2024-03-29T11:31:24.112 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.116+0000 7fad1bfff640 1 -- 172.21.15.135:0/1578851905 learned_addr learned my addr 172.21.15.135:0/1578851905 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:31:24.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.116+0000 7fad07fff640 10 client.?.objecter ms_handle_connect 0x7fad14007cd0 2024-03-29T11:31:24.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.117+0000 7fad07fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:24.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.117+0000 7fad07fff640 1 -- 172.21.15.135:0/1578851905 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158200985 0 0) 0x7fad14161100 con 0x7fad14007cd0 2024-03-29T11:31:24.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.117+0000 7fad07fff640 10 monclient(hunting): _negotiate 2024-03-29T11:31:24.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.117+0000 7fad07fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:24.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.117+0000 7fad07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:24.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.117+0000 7fad07fff640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:31:24.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.117+0000 7fad07fff640 1 -- 172.21.15.135:0/1578851905 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad00004eb0 con 0x7fad14007cd0 2024-03-29T11:31:24.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.117+0000 7fad07fff640 1 -- 172.21.15.135:0/1578851905 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3557558658 0 0) 0x7fad00004eb0 con 0x7fad14007cd0 2024-03-29T11:31:24.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.117+0000 7fad07fff640 1 -- 172.21.15.135:0/1578851905 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad14161100 con 0x7fad14007cd0 2024-03-29T11:31:24.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.117+0000 7fad07fff640 1 -- 172.21.15.135:0/1578851905 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fad08001bc0 con 0x7fad14007cd0 2024-03-29T11:31:24.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.117+0000 7fad07fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:24.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.117+0000 7fad07fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:24.113 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.117+0000 7fad07fff640 10 monclient(hunting): dump: 2024-03-29T11:31:24.113 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:24.114 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:24.114 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:24.114 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:24.114 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:24.114 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:24.114 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:24.114 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:24.114 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:24.114 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:24.114 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.117+0000 7fad07fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:24.114 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.117+0000 7fad07fff640 1 -- 172.21.15.135:0/1578851905 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2361914499 0 0) 0x7fad14161100 con 0x7fad14007cd0 2024-03-29T11:31:24.114 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.117+0000 7fad07fff640 1 -- 172.21.15.135:0/1578851905 >> v1:172.21.15.194:6789/0 conn(0x7fad140c3070 legacy=0x7fad14008a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:24.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.120+0000 7fad07fff640 1 -- 172.21.15.135:0/1578851905 >> v1:172.21.15.135:6789/0 conn(0x7fad14036a60 legacy=0x7fad14161620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:24.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.120+0000 7fad07fff640 10 monclient: _finish_hunting 0 2024-03-29T11:31:24.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.120+0000 7fad07fff640 1 monclient: found mon.c 2024-03-29T11:31:24.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.120+0000 7fad07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:24.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.120+0000 7fad07fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:24.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.120+0000 7fad07fff640 1 -- 172.21.15.135:0/1578851905 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad14009180 con 0x7fad14007cd0 2024-03-29T11:31:24.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.120+0000 7fad07fff640 10 monclient: _finish_auth 0 2024-03-29T11:31:24.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.120+0000 7fad07fff640 10 monclient: _check_auth_tickets 2024-03-29T11:31:24.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.120+0000 7fad07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:24.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.120+0000 7fad07fff640 10 client.?.objecter ms_handle_connect 0x7fad14036a60 2024-03-29T11:31:24.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.120+0000 7fad07fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:24.122 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.121+0000 7fad07fff640 1 -- 172.21.15.135:0/1578851905 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad08004c40 con 0x7fad14007cd0 2024-03-29T11:31:24.122 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.121+0000 7fad07fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:24.122 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.121+0000 7fad07fff640 1 -- 172.21.15.135:0/1578851905 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fad08004f50 con 0x7fad14007cd0 2024-03-29T11:31:24.122 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.121+0000 7fad07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:24.122 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.121+0000 7fad07fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:24.122 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.121+0000 7fad07fff640 10 monclient: dump: 2024-03-29T11:31:24.122 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:24.122 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:24.122 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:24.122 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:24.123 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:24.123 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:24.123 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:24.123 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:24.123 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:24.123 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:24.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.122+0000 7fad19ffb640 4 set_mon_vals no callback set 2024-03-29T11:31:24.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.123+0000 7fad208ac640 5 monclient: authenticate success, global_id 4490 2024-03-29T11:31:24.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.123+0000 7fad208ac640 10 monclient: _renew_subs 2024-03-29T11:31:24.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.123+0000 7fad208ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:24.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.123+0000 7fad208ac640 1 -- 172.21.15.135:0/1578851905 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad14038c60 con 0x7fad14007cd0 2024-03-29T11:31:24.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.123+0000 7fad208ac640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:24.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.123+0000 7fad208ac640 10 monclient: _renew_subs 2024-03-29T11:31:24.124 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.123+0000 7fad208ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:24.124 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.123+0000 7fad208ac640 1 -- 172.21.15.135:0/1578851905 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad1416cfc0 con 0x7fad14007cd0 2024-03-29T11:31:24.124 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.123+0000 7fad208ac640 1 librados: init done 2024-03-29T11:31:24.124 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.125+0000 7fad208ac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:24.124 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.125+0000 7fad208ac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:24.124 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.125+0000 7fad208ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:24.124 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.125+0000 7fad208ac640 1 -- 172.21.15.135:0/1578851905 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad14161290 con 0x7fad14007cd0 2024-03-29T11:31:24.125 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.130+0000 7fad07fff640 1 -- 172.21.15.135:0/1578851905 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fad0801af20 con 0x7fad14007cd0 2024-03-29T11:31:24.125 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.131+0000 7fad07fff640 1 -- 172.21.15.135:0/1578851905 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7fad08084b30 con 0x7fad14007cd0 2024-03-29T11:31:24.125 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.131+0000 7fad07fff640 10 client.4490.objecter ms_dispatch 0x7fad1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:31:24.126 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.131+0000 7fad07fff640 3 client.4490.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:31:24.126 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.131+0000 7fad07fff640 3 client.4490.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:31:24.126 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.131+0000 7fad07fff640 20 client.4490.objecter dump_active .. 0 homeless 2024-03-29T11:31:24.127 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.133+0000 7fad1b7fe640 10 monclient: get_auth_request con 0x7fad0006dc70 auth_method 0 2024-03-29T11:31:24.127 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.133+0000 7fad07fff640 10 client.4490.objecter ms_handle_connect 0x7fad0006dc70 2024-03-29T11:31:24.132 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.138+0000 7fad07fff640 1 -- 172.21.15.135:0/1578851905 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fad08050410 con 0x7fad14007cd0 2024-03-29T11:31:24.132 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.138+0000 7fad07fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:24.132 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.138+0000 7fad07fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:24.466 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.471+0000 7fad208ac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:24.467 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.471+0000 7fad208ac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:24.467 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.471+0000 7fad208ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:24.467 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.471+0000 7fad208ac640 1 -- 172.21.15.135:0/1578851905 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad14038df0 con 0x7fad14007cd0 2024-03-29T11:31:24.467 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.473+0000 7fad07fff640 1 -- 172.21.15.135:0/1578851905 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7fad08052340 con 0x7fad14007cd0 2024-03-29T11:31:24.467 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.473+0000 7fad07fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:24.467 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.473+0000 7fad07fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:24.468 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:31:24.468 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:24.470 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.476+0000 7fad208ac640 10 librados: watch_flush enter 2024-03-29T11:31:24.471 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.477+0000 7fad208ac640 10 librados: watch_flush exit 2024-03-29T11:31:24.471 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.477+0000 7fad208ac640 20 client.4490.objecter shutdown clearing up homeless session... 2024-03-29T11:31:24.472 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.477+0000 7fad208ac640 10 client.4490.objecter successfully canceled tick 2024-03-29T11:31:24.472 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.478+0000 7fad208ac640 1 -- 172.21.15.135:0/1578851905 >> v1:172.21.15.135:6816/3469875285 conn(0x7fad0006dc70 legacy=0x7fad00070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:24.473 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.478+0000 7fad208ac640 10 monclient: shutdown 2024-03-29T11:31:24.473 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.478+0000 7fad208ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:24.473 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.478+0000 7fad208ac640 1 -- 172.21.15.135:0/1578851905 >> v1:172.21.15.135:6790/0 conn(0x7fad14007cd0 legacy=0x7fad1416b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:24.476 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.481+0000 7fad208ac640 1 -- 172.21.15.135:0/1578851905 shutdown_connections 2024-03-29T11:31:24.476 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.481+0000 7fad208ac640 1 -- 172.21.15.135:0/1578851905 >> 172.21.15.135:0/1578851905 conn(0x7fad14005c00 msgr2=0x7fad140c04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:24.476 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.481+0000 7fad208ac640 1 -- 172.21.15.135:0/1578851905 shutdown_connections 2024-03-29T11:31:24.476 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.481+0000 7fad208ac640 1 -- 172.21.15.135:0/1578851905 wait complete. 2024-03-29T11:31:24.476 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:24.481+0000 7fad208ac640 1 librados: shutdown 2024-03-29T11:31:24.493 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:31.301 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:31.456 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.461+0000 7f482be25640 10 monclient: get_monmap_and_config 2024-03-29T11:31:31.456 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.461+0000 7f482be25640 10 monclient: build_initial_monmap 2024-03-29T11:31:31.456 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.461+0000 7f482be25640 10 monclient: monmap: 2024-03-29T11:31:31.456 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:31:31.456 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:31.457 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:31:31.462094+0000 2024-03-29T11:31:31.457 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:31:31.462094+0000 2024-03-29T11:31:31.457 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:31.457 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:31.457 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:31:31.457 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:31:31.457 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:31:31.457 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:31.457 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.463+0000 7f482be25640 1 Processor -- start 2024-03-29T11:31:31.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.467+0000 7f482be25640 1 -- start start 2024-03-29T11:31:31.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.467+0000 7f482be25640 10 monclient: init 2024-03-29T11:31:31.462 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.468+0000 7f482be25640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:31.462 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.468+0000 7f482be25640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:31:31.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.468+0000 7f482be25640 10 monclient(hunting): picked mon.noname-a con 0x7f48240a9ac0 addr v1:172.21.15.135:6789/0 2024-03-29T11:31:31.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.470+0000 7f482be25640 10 monclient(hunting): picked mon.noname-b con 0x7f482413ee90 addr v1:172.21.15.135:6790/0 2024-03-29T11:31:31.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.470+0000 7f482be25640 10 monclient(hunting): picked mon.noname-c con 0x7f48240af900 addr v1:172.21.15.194:6789/0 2024-03-29T11:31:31.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.470+0000 7f482be25640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48240afd10 con 0x7f48240a9ac0 2024-03-29T11:31:31.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.470+0000 7f482be25640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48240af6f0 con 0x7f48240af900 2024-03-29T11:31:31.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.470+0000 7f482be25640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48240aac80 con 0x7f482413ee90 2024-03-29T11:31:31.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.470+0000 7f482be25640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:31.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.470+0000 7f482be25640 10 monclient(hunting): authenticate will time out at 1198.838162s 2024-03-29T11:31:31.467 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.474+0000 7f4829bc1640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f48240a9ac0 0x7f48240adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:53065/0 (socket says 172.21.15.135:53065) 2024-03-29T11:31:31.468 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.474+0000 7f4829bc1640 1 -- 172.21.15.135:0/2215209483 learned_addr learned my addr 172.21.15.135:0/2215209483 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:31:31.469 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.474+0000 7f4828bbf640 1 -- 172.21.15.135:0/2215209483 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2834101371 0 0) 0x7f48240afd10 con 0x7f48240a9ac0 2024-03-29T11:31:31.469 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.474+0000 7f4828bbf640 10 monclient(hunting): _negotiate 2024-03-29T11:31:31.469 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.474+0000 7f4828bbf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:31.469 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.474+0000 7f4828bbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:31.469 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.474+0000 7f4828bbf640 10 monclient(hunting): my global_id is 4473 2024-03-29T11:31:31.469 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.474+0000 7f4828bbf640 1 -- 172.21.15.135:0/2215209483 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4810003520 con 0x7f48240a9ac0 2024-03-29T11:31:31.472 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.476+0000 7f4828bbf640 1 -- 172.21.15.135:0/2215209483 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 137512640 0 0) 0x7f4810003520 con 0x7f48240a9ac0 2024-03-29T11:31:31.473 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.476+0000 7f4828bbf640 1 -- 172.21.15.135:0/2215209483 >> v1:172.21.15.135:6790/0 conn(0x7f482413ee90 legacy=0x7f482413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:31.473 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.476+0000 7f4828bbf640 1 -- 172.21.15.135:0/2215209483 >> v1:172.21.15.194:6789/0 conn(0x7f48240af900 legacy=0x7f4824149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:31:31.473 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.476+0000 7f4828bbf640 10 monclient: _finish_hunting 0 2024-03-29T11:31:31.473 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.476+0000 7f4828bbf640 1 monclient: found mon.noname-a 2024-03-29T11:31:31.473 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.476+0000 7f4828bbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:31.473 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.476+0000 7f4828bbf640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.135:6789/0 2024-03-29T11:31:31.473 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.476+0000 7f4828bbf640 1 -- 172.21.15.135:0/2215209483 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48240aaf60 con 0x7f48240a9ac0 2024-03-29T11:31:31.473 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.476+0000 7f4828bbf640 10 monclient: _finish_auth 0 2024-03-29T11:31:31.473 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.476+0000 7f4828bbf640 10 monclient: _check_auth_tickets 2024-03-29T11:31:31.473 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.476+0000 7f4828bbf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:31.473 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.476+0000 7f4828bbf640 1 -- 172.21.15.135:0/2215209483 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f4820001690 con 0x7f48240a9ac0 2024-03-29T11:31:31.473 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.476+0000 7f4828bbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:31.473 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.476+0000 7f4828bbf640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:31:31.474 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.476+0000 7f4828bbf640 10 monclient: dump: 2024-03-29T11:31:31.474 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:31.474 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:31.474 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:31.474 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:31.474 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:31.474 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:31.474 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:31.474 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:31.474 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:31.474 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:31.474 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.476+0000 7f482be25640 5 monclient: authenticate success, global_id 4473 2024-03-29T11:31:31.475 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.476+0000 7f482be25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:31.475 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.476+0000 7f4828bbf640 1 -- 172.21.15.135:0/2215209483 <== mon.0 v1:172.21.15.135:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4820001860 con 0x7f48240a9ac0 2024-03-29T11:31:31.475 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.476+0000 7f4828bbf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:31.475 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.476+0000 7f4828bbf640 1 -- 172.21.15.135:0/2215209483 <== mon.0 v1:172.21.15.135:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f4820001cb0 con 0x7f48240a9ac0 2024-03-29T11:31:31.475 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.476+0000 7f4828bbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:31.475 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.476+0000 7f4828bbf640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:31:31.475 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.476+0000 7f4828bbf640 10 monclient: dump: 2024-03-29T11:31:31.475 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:31.475 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:31.475 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:31.475 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:31.475 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:31.476 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:31.476 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:31.476 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:31.476 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:31.476 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:31.476 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.476+0000 7f482be25640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:31.476 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.476+0000 7f482be25640 4 set_mon_vals no callback set 2024-03-29T11:31:31.476 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.478+0000 7f482be25640 10 monclient: shutdown 2024-03-29T11:31:31.476 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.478+0000 7f482be25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:31.476 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.478+0000 7f482be25640 1 -- 172.21.15.135:0/2215209483 >> v1:172.21.15.135:6789/0 conn(0x7f48240a9ac0 legacy=0x7f48240adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:31.476 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.478+0000 7f482be25640 1 -- 172.21.15.135:0/2215209483 shutdown_connections 2024-03-29T11:31:31.476 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.478+0000 7f482be25640 1 -- 172.21.15.135:0/2215209483 >> 172.21.15.135:0/2215209483 conn(0x7f48240b35e0 msgr2=0x7f48240b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:31.479 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.481+0000 7f482be25640 1 -- 172.21.15.135:0/2215209483 shutdown_connections 2024-03-29T11:31:31.479 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.481+0000 7f482be25640 1 -- 172.21.15.135:0/2215209483 wait complete. 2024-03-29T11:31:31.479 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.482+0000 7f482be25640 10 monclient: build_initial_monmap 2024-03-29T11:31:31.479 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.482+0000 7f482be25640 10 monclient: monmap: 2024-03-29T11:31:31.479 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:31:31.479 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:31.479 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:31:31.479 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:31:31.479 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:31.479 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:31.479 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:31:31.479 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:31:31.480 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:31:31.480 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:31.480 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.482+0000 7f482be25640 1 librados: starting msgr at 2024-03-29T11:31:31.480 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.482+0000 7f482be25640 1 librados: starting objecter 2024-03-29T11:31:31.480 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.483+0000 7f482be25640 1 Processor -- start 2024-03-29T11:31:31.480 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.483+0000 7f482be25640 1 -- start start 2024-03-29T11:31:31.480 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.483+0000 7f482be25640 1 librados: setting wanted keys 2024-03-29T11:31:31.480 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.483+0000 7f482be25640 1 librados: calling monclient init 2024-03-29T11:31:31.480 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.483+0000 7f482be25640 10 monclient: init 2024-03-29T11:31:31.480 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.484+0000 7f482be25640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:31.480 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.484+0000 7f482be25640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:31:31.480 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.484+0000 7f482be25640 10 monclient(hunting): picked mon.noname-b con 0x7f48240a9ac0 addr v1:172.21.15.194:6789/0 2024-03-29T11:31:31.481 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.487+0000 7f482be25640 10 monclient(hunting): picked mon.noname-a con 0x7f48240af900 addr v1:172.21.15.135:6789/0 2024-03-29T11:31:31.484 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.487+0000 7f482be25640 10 monclient(hunting): picked mon.noname-c con 0x7f482413ee90 addr v1:172.21.15.135:6790/0 2024-03-29T11:31:31.484 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.490+0000 7f4829bc1640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f48240a9ac0 0x7f48241d6530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:44421/0 (socket says 172.21.15.135:44421) 2024-03-29T11:31:31.484 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.490+0000 7f4829bc1640 1 -- 172.21.15.135:0/2618807490 learned_addr learned my addr 172.21.15.135:0/2618807490 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:31:31.484 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.490+0000 7f482a3c2640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f482413ee90 0x7f482406e9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:39595/0 (socket says 172.21.15.135:39595) 2024-03-29T11:31:31.484 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.490+0000 7f482be25640 1 -- 172.21.15.135:0/2618807490 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48241d70b0 con 0x7f48240af900 2024-03-29T11:31:31.485 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.490+0000 7f482be25640 1 -- 172.21.15.135:0/2618807490 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48241d6c40 con 0x7f48240a9ac0 2024-03-29T11:31:31.485 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.490+0000 7f482be25640 1 -- 172.21.15.135:0/2618807490 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48241d6dd0 con 0x7f482413ee90 2024-03-29T11:31:31.485 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.490+0000 7f482be25640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:31.485 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.491+0000 7f482be25640 10 monclient(hunting): authenticate will time out at 1198.859112s 2024-03-29T11:31:31.489 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.491+0000 7f481a7fc640 10 client.?.objecter ms_handle_connect 0x7f482413ee90 2024-03-29T11:31:31.489 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.491+0000 7f481a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:31.489 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.491+0000 7f481a7fc640 10 client.?.objecter ms_handle_connect 0x7f48240a9ac0 2024-03-29T11:31:31.489 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.491+0000 7f481a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:31.489 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.491+0000 7f481a7fc640 1 -- 172.21.15.135:0/2618807490 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1662502689 0 0) 0x7f48241d6c40 con 0x7f48240a9ac0 2024-03-29T11:31:31.489 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.491+0000 7f481a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:31:31.489 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.491+0000 7f481a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:31.489 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.491+0000 7f481a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:31.489 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.491+0000 7f481a7fc640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:31:31.489 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.491+0000 7f481a7fc640 1 -- 172.21.15.135:0/2618807490 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4808004eb0 con 0x7f48240a9ac0 2024-03-29T11:31:31.489 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.491+0000 7f481a7fc640 1 -- 172.21.15.135:0/2618807490 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3992320569 0 0) 0x7f48241d6dd0 con 0x7f482413ee90 2024-03-29T11:31:31.489 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.491+0000 7f481a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:31:31.489 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.491+0000 7f481a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:31.490 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.491+0000 7f481a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:31.490 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.491+0000 7f481a7fc640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:31:31.490 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.491+0000 7f481a7fc640 1 -- 172.21.15.135:0/2618807490 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48241d6c40 con 0x7f482413ee90 2024-03-29T11:31:31.490 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.492+0000 7f481a7fc640 1 -- 172.21.15.135:0/2618807490 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2086364770 0 0) 0x7f48241d6c40 con 0x7f482413ee90 2024-03-29T11:31:31.490 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.492+0000 7f481a7fc640 1 -- 172.21.15.135:0/2618807490 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48241d6dd0 con 0x7f482413ee90 2024-03-29T11:31:31.490 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.492+0000 7f481a7fc640 1 -- 172.21.15.135:0/2618807490 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f481c001b30 con 0x7f482413ee90 2024-03-29T11:31:31.490 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.492+0000 7f481a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:31.490 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.492+0000 7f481a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:31.490 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.492+0000 7f481a7fc640 10 monclient(hunting): dump: 2024-03-29T11:31:31.490 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:31.490 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:31.490 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:31.490 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:31.490 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:31.491 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:31.491 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:31.491 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:31.491 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:31.491 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:31.491 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.492+0000 7f481a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:31.491 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.492+0000 7f481a7fc640 1 -- 172.21.15.135:0/2618807490 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2639285704 0 0) 0x7f48241d6dd0 con 0x7f482413ee90 2024-03-29T11:31:31.491 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.492+0000 7f481a7fc640 1 -- 172.21.15.135:0/2618807490 >> v1:172.21.15.194:6789/0 conn(0x7f48240a9ac0 legacy=0x7f48241d6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:31.491 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.492+0000 7f481a7fc640 1 -- 172.21.15.135:0/2618807490 >> v1:172.21.15.135:6789/0 conn(0x7f48240af900 legacy=0x7f48241398f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:31.491 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.492+0000 7f481a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:31:31.491 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.492+0000 7f481a7fc640 1 monclient: found mon.c 2024-03-29T11:31:31.491 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.492+0000 7f481a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:31.491 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.492+0000 7f481a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:31.491 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.492+0000 7f481a7fc640 1 -- 172.21.15.135:0/2618807490 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f482413a000 con 0x7f482413ee90 2024-03-29T11:31:31.495 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.492+0000 7f481a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:31:31.495 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.492+0000 7f481a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:31:31.495 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.492+0000 7f481a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:31.495 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.492+0000 7f481a7fc640 1 -- 172.21.15.135:0/2618807490 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f481c002040 con 0x7f482413ee90 2024-03-29T11:31:31.495 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.493+0000 7f481a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:31.495 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.493+0000 7f481a7fc640 1 -- 172.21.15.135:0/2618807490 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f481c004d60 con 0x7f482413ee90 2024-03-29T11:31:31.495 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.493+0000 7f481a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:31.495 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.493+0000 7f481b7fe640 4 set_mon_vals no callback set 2024-03-29T11:31:31.496 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.493+0000 7f481a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:31.496 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.493+0000 7f481a7fc640 10 monclient: dump: 2024-03-29T11:31:31.496 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:31.496 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:31.496 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:31.496 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:31.496 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:31.496 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:31.496 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:31.496 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:31.496 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:31.496 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:31.496 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.494+0000 7f482be25640 5 monclient: authenticate success, global_id 4493 2024-03-29T11:31:31.496 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.494+0000 7f482be25640 10 monclient: _renew_subs 2024-03-29T11:31:31.497 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.494+0000 7f482be25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:31.497 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.494+0000 7f482be25640 1 -- 172.21.15.135:0/2618807490 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f482413a250 con 0x7f482413ee90 2024-03-29T11:31:31.497 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.494+0000 7f482be25640 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:31.497 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.494+0000 7f482be25640 10 monclient: _renew_subs 2024-03-29T11:31:31.497 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.494+0000 7f482be25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:31.497 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.494+0000 7f482be25640 1 -- 172.21.15.135:0/2618807490 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4824142370 con 0x7f482413ee90 2024-03-29T11:31:31.497 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.494+0000 7f482be25640 1 librados: init done 2024-03-29T11:31:31.497 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.498+0000 7f4807fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:31.497 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.498+0000 7f4807fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:31.497 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.498+0000 7f4807fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:31.497 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.498+0000 7f4807fff640 1 -- 172.21.15.135:0/2618807490 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4824075f10 con 0x7f482413ee90 2024-03-29T11:31:31.497 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.500+0000 7f481a7fc640 1 -- 172.21.15.135:0/2618807490 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f481c01ad30 con 0x7f482413ee90 2024-03-29T11:31:31.497 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.501+0000 7f481a7fc640 1 -- 172.21.15.135:0/2618807490 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f481c084c40 con 0x7f482413ee90 2024-03-29T11:31:31.497 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.501+0000 7f481a7fc640 10 client.4493.objecter ms_dispatch 0x7f4824000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:31:31.497 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.501+0000 7f481a7fc640 3 client.4493.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:31:31.498 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.501+0000 7f481a7fc640 3 client.4493.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:31:31.498 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.501+0000 7f481a7fc640 20 client.4493.objecter dump_active .. 0 homeless 2024-03-29T11:31:31.498 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.501+0000 7f4829bc1640 10 monclient: get_auth_request con 0x7f480807ec50 auth_method 0 2024-03-29T11:31:31.498 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.502+0000 7f481a7fc640 10 client.4493.objecter ms_handle_connect 0x7f480807ec50 2024-03-29T11:31:31.504 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.507+0000 7f481a7fc640 1 -- 172.21.15.135:0/2618807490 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f481c050520 con 0x7f482413ee90 2024-03-29T11:31:31.504 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.507+0000 7f481a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:31.504 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.507+0000 7f481a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:31.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.832+0000 7f4807fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:31.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.832+0000 7f4807fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:31.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.832+0000 7f4807fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:31.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.832+0000 7f4807fff640 1 -- 172.21.15.135:0/2618807490 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48240abcf0 con 0x7f482413ee90 2024-03-29T11:31:31.829 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.836+0000 7f481a7fc640 1 -- 172.21.15.135:0/2618807490 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f481c052450 con 0x7f482413ee90 2024-03-29T11:31:31.829 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.836+0000 7f481a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:31.829 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.836+0000 7f481a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:31.830 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:31:31.830 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:31.833 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.839+0000 7f4807fff640 10 librados: watch_flush enter 2024-03-29T11:31:31.833 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.839+0000 7f4807fff640 10 librados: watch_flush exit 2024-03-29T11:31:31.833 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.839+0000 7f4807fff640 20 client.4493.objecter shutdown clearing up homeless session... 2024-03-29T11:31:31.834 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.839+0000 7f4807fff640 10 client.4493.objecter successfully canceled tick 2024-03-29T11:31:31.834 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.839+0000 7f4807fff640 1 -- 172.21.15.135:0/2618807490 >> v1:172.21.15.135:6816/3469875285 conn(0x7f480807ec50 legacy=0x7f4808081110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:31.834 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.839+0000 7f4807fff640 10 monclient: shutdown 2024-03-29T11:31:31.834 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.839+0000 7f4807fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:31.834 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.839+0000 7f4807fff640 1 -- 172.21.15.135:0/2618807490 >> v1:172.21.15.135:6790/0 conn(0x7f482413ee90 legacy=0x7f482406e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:31.834 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.839+0000 7f4807fff640 1 -- 172.21.15.135:0/2618807490 shutdown_connections 2024-03-29T11:31:31.834 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.839+0000 7f4807fff640 1 -- 172.21.15.135:0/2618807490 >> 172.21.15.135:0/2618807490 conn(0x7f48240b35e0 msgr2=0x7f482413b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:31.835 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.841+0000 7f4807fff640 1 -- 172.21.15.135:0/2618807490 shutdown_connections 2024-03-29T11:31:31.835 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.841+0000 7f4807fff640 1 -- 172.21.15.135:0/2618807490 wait complete. 2024-03-29T11:31:31.835 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:31.841+0000 7f4807fff640 1 librados: shutdown 2024-03-29T11:31:31.852 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:38.659 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:38.811 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.817+0000 7f18bb579640 10 monclient: get_monmap_and_config 2024-03-29T11:31:38.812 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.817+0000 7f18bb579640 10 monclient: build_initial_monmap 2024-03-29T11:31:38.812 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.817+0000 7f18bb579640 10 monclient: monmap: 2024-03-29T11:31:38.812 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:31:38.812 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:38.812 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:31:38.818198+0000 2024-03-29T11:31:38.812 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:31:38.818198+0000 2024-03-29T11:31:38.812 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:38.812 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:38.812 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:31:38.812 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:31:38.812 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:31:38.812 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:38.816 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.819+0000 7f18bb579640 1 Processor -- start 2024-03-29T11:31:38.816 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.819+0000 7f18bb579640 1 -- start start 2024-03-29T11:31:38.816 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.819+0000 7f18bb579640 10 monclient: init 2024-03-29T11:31:38.816 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.820+0000 7f18bb579640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:38.816 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.820+0000 7f18bb579640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:31:38.816 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.822+0000 7f18bb579640 10 monclient(hunting): picked mon.noname-a con 0x7f18ac006890 addr v1:172.21.15.135:6789/0 2024-03-29T11:31:38.816 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.822+0000 7f18bb579640 10 monclient(hunting): picked mon.noname-c con 0x7f18ac006c70 addr v1:172.21.15.194:6789/0 2024-03-29T11:31:38.816 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.822+0000 7f18bb579640 10 monclient(hunting): picked mon.noname-b con 0x7f18ac02b4c0 addr v1:172.21.15.135:6790/0 2024-03-29T11:31:38.816 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.822+0000 7f18bb579640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ac02c830 con 0x7f18ac006890 2024-03-29T11:31:38.816 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.822+0000 7f18bb579640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ac02afa0 con 0x7f18ac006c70 2024-03-29T11:31:38.816 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.822+0000 7f18bb579640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ac02aba0 con 0x7f18ac02b4c0 2024-03-29T11:31:38.816 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.822+0000 7f18bb579640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:38.816 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.822+0000 7f18bb579640 10 monclient(hunting): authenticate will time out at 1206.190152s 2024-03-29T11:31:38.819 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.824+0000 7f18ba577640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f18ac006890 0x7f18ac0c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:38653/0 (socket says 172.21.15.135:38653) 2024-03-29T11:31:38.819 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.824+0000 7f18ba577640 1 -- 172.21.15.135:0/2864314407 learned_addr learned my addr 172.21.15.135:0/2864314407 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:31:38.820 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.826+0000 7f18b9575640 1 -- 172.21.15.135:0/2864314407 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172804362 0 0) 0x7f18ac02c830 con 0x7f18ac006890 2024-03-29T11:31:38.820 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18b9575640 10 monclient(hunting): _negotiate 2024-03-29T11:31:38.820 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18b9575640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:38.820 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18b9575640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:38.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18b9575640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:31:38.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18b9575640 1 -- 172.21.15.135:0/2864314407 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18a0003520 con 0x7f18ac006890 2024-03-29T11:31:38.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18b9575640 1 -- 172.21.15.135:0/2864314407 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2868897678 0 0) 0x7f18a0003520 con 0x7f18ac006890 2024-03-29T11:31:38.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18b9575640 1 -- 172.21.15.135:0/2864314407 >> v1:172.21.15.135:6790/0 conn(0x7f18ac02b4c0 legacy=0x7f18ac0cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:38.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18b9575640 1 -- 172.21.15.135:0/2864314407 >> v1:172.21.15.194:6789/0 conn(0x7f18ac006c70 legacy=0x7f18ac0c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:38.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18b9575640 10 monclient: _finish_hunting 0 2024-03-29T11:31:38.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18b9575640 1 monclient: found mon.noname-a 2024-03-29T11:31:38.821 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18b9575640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:38.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18b9575640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.135:6789/0 2024-03-29T11:31:38.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18b9575640 1 -- 172.21.15.135:0/2864314407 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ac02ad30 con 0x7f18ac006890 2024-03-29T11:31:38.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18b9575640 10 monclient: _finish_auth 0 2024-03-29T11:31:38.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18b9575640 10 monclient: _check_auth_tickets 2024-03-29T11:31:38.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18b9575640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:38.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18b9575640 1 -- 172.21.15.135:0/2864314407 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f18b404f730 con 0x7f18ac006890 2024-03-29T11:31:38.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18b9575640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:38.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18b9575640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:31:38.822 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18b9575640 10 monclient: dump: 2024-03-29T11:31:38.822 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:38.822 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:38.822 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:38.822 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:38.822 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:38.823 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:38.823 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:38.823 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:38.823 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:38.823 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:38.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18b9575640 1 -- 172.21.15.135:0/2864314407 <== mon.0 v1:172.21.15.135:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18b4075f10 con 0x7f18ac006890 2024-03-29T11:31:38.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18b9575640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:38.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18b9575640 1 -- 172.21.15.135:0/2864314407 <== mon.0 v1:172.21.15.135:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f18b40758b0 con 0x7f18ac006890 2024-03-29T11:31:38.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18b9575640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:38.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18b9575640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:31:38.823 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18b9575640 10 monclient: dump: 2024-03-29T11:31:38.823 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:38.823 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:38.823 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:38.823 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:38.824 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:38.824 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:38.824 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:38.824 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:38.824 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:38.824 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:38.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18bb579640 5 monclient: authenticate success, global_id 4476 2024-03-29T11:31:38.825 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18bb579640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:38.825 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.827+0000 7f18bb579640 4 set_mon_vals no callback set 2024-03-29T11:31:38.825 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.828+0000 7f18bb579640 10 monclient: shutdown 2024-03-29T11:31:38.825 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.828+0000 7f18bb579640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:38.825 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.828+0000 7f18bb579640 1 -- 172.21.15.135:0/2864314407 >> v1:172.21.15.135:6789/0 conn(0x7f18ac006890 legacy=0x7f18ac0c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:38.825 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.829+0000 7f18bb579640 1 -- 172.21.15.135:0/2864314407 shutdown_connections 2024-03-29T11:31:38.825 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.829+0000 7f18bb579640 1 -- 172.21.15.135:0/2864314407 >> 172.21.15.135:0/2864314407 conn(0x7f18ac0058f0 msgr2=0x7f18ac005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:38.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.833+0000 7f18bb579640 1 -- 172.21.15.135:0/2864314407 shutdown_connections 2024-03-29T11:31:38.828 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.834+0000 7f18bb579640 1 -- 172.21.15.135:0/2864314407 wait complete. 2024-03-29T11:31:38.830 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.835+0000 7f18bb579640 10 monclient: build_initial_monmap 2024-03-29T11:31:38.830 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.835+0000 7f18bb579640 10 monclient: monmap: 2024-03-29T11:31:38.830 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:31:38.830 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:38.830 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:31:38.830 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:31:38.830 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:38.830 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:38.830 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:31:38.831 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:31:38.831 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:31:38.831 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:38.831 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.835+0000 7f18bb579640 1 librados: starting msgr at 2024-03-29T11:31:38.831 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.835+0000 7f18bb579640 1 librados: starting objecter 2024-03-29T11:31:38.831 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.835+0000 7f18bb579640 1 Processor -- start 2024-03-29T11:31:38.831 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.836+0000 7f18bb579640 1 -- start start 2024-03-29T11:31:38.831 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.836+0000 7f18bb579640 1 librados: setting wanted keys 2024-03-29T11:31:38.831 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.836+0000 7f18bb579640 1 librados: calling monclient init 2024-03-29T11:31:38.831 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.836+0000 7f18bb579640 10 monclient: init 2024-03-29T11:31:38.832 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.838+0000 7f18bb579640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:38.832 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.838+0000 7f18bb579640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:31:38.832 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.838+0000 7f18bb579640 10 monclient(hunting): picked mon.noname-a con 0x7f18ac006890 addr v1:172.21.15.135:6789/0 2024-03-29T11:31:38.832 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.838+0000 7f18bb579640 10 monclient(hunting): picked mon.noname-b con 0x7f18ac02b4c0 addr v1:172.21.15.194:6789/0 2024-03-29T11:31:38.833 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.838+0000 7f18bb579640 10 monclient(hunting): picked mon.noname-c con 0x7f18ac0c3f70 addr v1:172.21.15.135:6790/0 2024-03-29T11:31:38.833 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.839+0000 7f18bb579640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ac02eca0 con 0x7f18ac006890 2024-03-29T11:31:38.833 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.839+0000 7f18bb579640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ac02fbe0 con 0x7f18ac02b4c0 2024-03-29T11:31:38.833 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.839+0000 7f18bb579640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18ac0c4400 con 0x7f18ac0c3f70 2024-03-29T11:31:38.833 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.839+0000 7f18bb579640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:38.833 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.839+0000 7f18bb579640 10 monclient(hunting): authenticate will time out at 1206.207432s 2024-03-29T11:31:38.835 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.842+0000 7f18ba577640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f18ac006890 0x7f18ac02e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:39121/0 (socket says 172.21.15.135:39121) 2024-03-29T11:31:38.836 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.842+0000 7f18ba577640 1 -- 172.21.15.135:0/2795694442 learned_addr learned my addr 172.21.15.135:0/2795694442 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:31:38.836 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.843+0000 7f18aaffd640 10 client.?.objecter ms_handle_connect 0x7f18ac006890 2024-03-29T11:31:38.837 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.843+0000 7f18aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:38.837 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.843+0000 7f18aaffd640 1 -- 172.21.15.135:0/2795694442 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4196729571 0 0) 0x7f18ac02eca0 con 0x7f18ac006890 2024-03-29T11:31:38.837 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.843+0000 7f18aaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:38.837 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.843+0000 7f18aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:38.837 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.843+0000 7f18aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:38.837 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.843+0000 7f18aaffd640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:31:38.843 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.843+0000 7f18aaffd640 1 -- 172.21.15.135:0/2795694442 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1898004eb0 con 0x7f18ac006890 2024-03-29T11:31:38.843 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.843+0000 7f18aaffd640 10 client.?.objecter ms_handle_connect 0x7f18ac0c3f70 2024-03-29T11:31:38.843 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.843+0000 7f18aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:38.843 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 10 client.?.objecter ms_handle_connect 0x7f18ac02b4c0 2024-03-29T11:31:38.843 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:38.843 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 1 -- 172.21.15.135:0/2795694442 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2133114304 0 0) 0x7f1898004eb0 con 0x7f18ac006890 2024-03-29T11:31:38.843 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 1 -- 172.21.15.135:0/2795694442 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18ac02eca0 con 0x7f18ac006890 2024-03-29T11:31:38.843 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 1 -- 172.21.15.135:0/2795694442 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f18b404f400 con 0x7f18ac006890 2024-03-29T11:31:38.843 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:38.843 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:31:38.843 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 10 monclient(hunting): dump: 2024-03-29T11:31:38.844 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:38.844 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:38.844 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:38.844 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:38.844 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:38.844 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:38.844 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:38.844 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:38.844 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:38.844 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:38.844 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:38.844 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 1 -- 172.21.15.135:0/2795694442 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 605158256 0 0) 0x7f18ac02fbe0 con 0x7f18ac02b4c0 2024-03-29T11:31:38.844 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:38.845 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:38.845 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:38.845 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:31:38.845 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 1 -- 172.21.15.135:0/2795694442 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1898004eb0 con 0x7f18ac02b4c0 2024-03-29T11:31:38.845 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 1 -- 172.21.15.135:0/2795694442 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013526895 0 0) 0x7f18ac0c4400 con 0x7f18ac0c3f70 2024-03-29T11:31:38.845 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:38.845 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:38.845 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:38.845 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:31:38.845 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 1 -- 172.21.15.135:0/2795694442 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1898005770 con 0x7f18ac0c3f70 2024-03-29T11:31:38.845 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 1 -- 172.21.15.135:0/2795694442 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3684766920 0 0) 0x7f18ac02eca0 con 0x7f18ac006890 2024-03-29T11:31:38.845 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 1 -- 172.21.15.135:0/2795694442 >> v1:172.21.15.135:6790/0 conn(0x7f18ac0c3f70 legacy=0x7f18ac165150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:38.845 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 1 -- 172.21.15.135:0/2795694442 >> v1:172.21.15.194:6789/0 conn(0x7f18ac02b4c0 legacy=0x7f18ac0305a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:38.845 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:31:38.845 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 1 monclient: found mon.a 2024-03-29T11:31:38.846 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:38.846 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:31:38.846 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 1 -- 172.21.15.135:0/2795694442 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ac030cb0 con 0x7f18ac006890 2024-03-29T11:31:38.846 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 10 monclient: _finish_auth 0 2024-03-29T11:31:38.846 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:31:38.846 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:38.846 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18bb579640 5 monclient: authenticate success, global_id 4479 2024-03-29T11:31:38.846 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18bb579640 10 monclient: _renew_subs 2024-03-29T11:31:38.846 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18bb579640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:31:38.846 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18bb579640 1 -- 172.21.15.135:0/2795694442 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18ac02fe30 con 0x7f18ac006890 2024-03-29T11:31:38.846 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18bb579640 10 client.4479.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:38.846 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18bb579640 10 monclient: _renew_subs 2024-03-29T11:31:38.846 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18bb579640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:31:38.847 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.844+0000 7f18bb579640 1 -- 172.21.15.135:0/2795694442 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18ac02dab0 con 0x7f18ac006890 2024-03-29T11:31:38.847 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.845+0000 7f18bb579640 1 librados: init done 2024-03-29T11:31:38.847 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.846+0000 7f18aaffd640 1 -- 172.21.15.135:0/2795694442 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18b404fdd0 con 0x7f18ac006890 2024-03-29T11:31:38.847 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.846+0000 7f18aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:38.847 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.846+0000 7f18aaffd640 1 -- 172.21.15.135:0/2795694442 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f18b4067bf0 con 0x7f18ac006890 2024-03-29T11:31:38.847 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.846+0000 7f18aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:38.847 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.846+0000 7f18aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:31:38.847 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.846+0000 7f18aaffd640 10 monclient: dump: 2024-03-29T11:31:38.847 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:38.847 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:38.847 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:38.847 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:38.847 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:38.847 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:38.847 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:38.848 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:38.848 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:38.848 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:38.848 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.846+0000 7f18bb579640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:38.848 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.846+0000 7f18bb579640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:38.848 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.846+0000 7f18bb579640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:31:38.848 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.846+0000 7f18b8d74640 4 set_mon_vals no callback set 2024-03-29T11:31:38.848 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.846+0000 7f18bb579640 1 -- 172.21.15.135:0/2795694442 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18ac02efd0 con 0x7f18ac006890 2024-03-29T11:31:38.848 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.847+0000 7f18aaffd640 1 -- 172.21.15.135:0/2795694442 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f18b4067e10 con 0x7f18ac006890 2024-03-29T11:31:38.848 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.848+0000 7f18aaffd640 1 -- 172.21.15.135:0/2795694442 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f18b40ff650 con 0x7f18ac006890 2024-03-29T11:31:38.848 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.848+0000 7f18aaffd640 10 client.4479.objecter ms_dispatch 0x7f18b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:31:38.848 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.848+0000 7f18aaffd640 3 client.4479.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:31:38.848 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.848+0000 7f18aaffd640 3 client.4479.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:31:38.848 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.848+0000 7f18aaffd640 20 client.4479.objecter dump_active .. 0 homeless 2024-03-29T11:31:38.849 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.848+0000 7f18b9d76640 10 monclient: get_auth_request con 0x7f189806dfb0 auth_method 0 2024-03-29T11:31:38.849 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.848+0000 7f18aaffd640 10 client.4479.objecter ms_handle_connect 0x7f189806dfb0 2024-03-29T11:31:38.849 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.855+0000 7f18aaffd640 1 -- 172.21.15.135:0/2795694442 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f18b40caf30 con 0x7f18ac006890 2024-03-29T11:31:38.849 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.855+0000 7f18aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:38.849 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:38.855+0000 7f18aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:39.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:39.246+0000 7f18bb579640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:39.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:39.246+0000 7f18bb579640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:39.241 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:39.246+0000 7f18bb579640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:31:39.241 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:39.246+0000 7f18bb579640 1 -- 172.21.15.135:0/2795694442 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18ac004cd0 con 0x7f18ac006890 2024-03-29T11:31:39.241 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:39.248+0000 7f18aaffd640 1 -- 172.21.15.135:0/2795694442 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f18b40cce60 con 0x7f18ac006890 2024-03-29T11:31:39.242 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:39.248+0000 7f18aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:39.242 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:39.248+0000 7f18aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:39.242 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:31:39.242 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:39.245 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:39.251+0000 7f18bb579640 10 librados: watch_flush enter 2024-03-29T11:31:39.245 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:39.252+0000 7f18bb579640 10 librados: watch_flush exit 2024-03-29T11:31:39.246 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:39.252+0000 7f18bb579640 20 client.4479.objecter shutdown clearing up homeless session... 2024-03-29T11:31:39.246 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:39.252+0000 7f18bb579640 10 client.4479.objecter successfully canceled tick 2024-03-29T11:31:39.246 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:39.252+0000 7f18bb579640 1 -- 172.21.15.135:0/2795694442 >> v1:172.21.15.135:6816/3469875285 conn(0x7f189806dfb0 legacy=0x7f1898070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:39.246 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:39.252+0000 7f18bb579640 10 monclient: shutdown 2024-03-29T11:31:39.246 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:39.252+0000 7f18bb579640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:39.246 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:39.252+0000 7f18bb579640 1 -- 172.21.15.135:0/2795694442 >> v1:172.21.15.135:6789/0 conn(0x7f18ac006890 legacy=0x7f18ac02e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:39.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:39.252+0000 7f18bb579640 1 -- 172.21.15.135:0/2795694442 shutdown_connections 2024-03-29T11:31:39.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:39.252+0000 7f18bb579640 1 -- 172.21.15.135:0/2795694442 >> 172.21.15.135:0/2795694442 conn(0x7f18ac0058f0 msgr2=0x7f18ac038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:39.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:39.252+0000 7f18bb579640 1 -- 172.21.15.135:0/2795694442 shutdown_connections 2024-03-29T11:31:39.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:39.252+0000 7f18bb579640 1 -- 172.21.15.135:0/2795694442 wait complete. 2024-03-29T11:31:39.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:39.253+0000 7f18bb579640 1 librados: shutdown 2024-03-29T11:31:39.267 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:46.074 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:46.224 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.230+0000 7f6d7e365640 10 monclient: get_monmap_and_config 2024-03-29T11:31:46.225 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.230+0000 7f6d7e365640 10 monclient: build_initial_monmap 2024-03-29T11:31:46.225 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.230+0000 7f6d7e365640 10 monclient: monmap: 2024-03-29T11:31:46.225 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:31:46.225 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:46.225 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:31:46.231413+0000 2024-03-29T11:31:46.225 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:31:46.231413+0000 2024-03-29T11:31:46.225 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:46.225 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:46.225 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:31:46.225 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:31:46.226 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:31:46.226 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:46.227 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.232+0000 7f6d7e365640 1 Processor -- start 2024-03-29T11:31:46.227 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.232+0000 7f6d7e365640 1 -- start start 2024-03-29T11:31:46.227 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.232+0000 7f6d7e365640 10 monclient: init 2024-03-29T11:31:46.228 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.234+0000 7f6d7e365640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:46.228 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.234+0000 7f6d7e365640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:31:46.228 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.234+0000 7f6d7e365640 10 monclient(hunting): picked mon.noname-b con 0x7f6d781457f0 addr v1:172.21.15.135:6790/0 2024-03-29T11:31:46.228 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.234+0000 7f6d7e365640 10 monclient(hunting): picked mon.noname-a con 0x7f6d7813f650 addr v1:172.21.15.135:6789/0 2024-03-29T11:31:46.228 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.234+0000 7f6d7e365640 10 monclient(hunting): picked mon.noname-c con 0x7f6d780a9d70 addr v1:172.21.15.194:6789/0 2024-03-29T11:31:46.229 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.234+0000 7f6d7e365640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d780aaef0 con 0x7f6d7813f650 2024-03-29T11:31:46.229 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.234+0000 7f6d7e365640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d78145bd0 con 0x7f6d780a9d70 2024-03-29T11:31:46.229 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.234+0000 7f6d7e365640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d78145eb0 con 0x7f6d781457f0 2024-03-29T11:31:46.229 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.234+0000 7f6d7e365640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:46.229 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.234+0000 7f6d7e365640 10 monclient(hunting): authenticate will time out at 1213.602163s 2024-03-29T11:31:46.229 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.234+0000 7f6d77fff640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f6d781457f0 0x7f6d7813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:54634/0 (socket says 172.21.15.135:54634) 2024-03-29T11:31:46.229 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.234+0000 7f6d77fff640 1 -- 172.21.15.135:0/545783659 learned_addr learned my addr 172.21.15.135:0/545783659 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:31:46.231 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.234+0000 7f6d76ffd640 1 -- 172.21.15.135:0/545783659 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2128132125 0 0) 0x7f6d78145eb0 con 0x7f6d781457f0 2024-03-29T11:31:46.231 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.234+0000 7f6d76ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:46.232 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.234+0000 7f6d76ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:46.232 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.234+0000 7f6d76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:46.232 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.234+0000 7f6d76ffd640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:31:46.232 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.235+0000 7f6d76ffd640 1 -- 172.21.15.135:0/545783659 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d60003520 con 0x7f6d781457f0 2024-03-29T11:31:46.232 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.235+0000 7f6d76ffd640 1 -- 172.21.15.135:0/545783659 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 846537708 0 0) 0x7f6d60003520 con 0x7f6d781457f0 2024-03-29T11:31:46.232 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.235+0000 7f6d76ffd640 1 -- 172.21.15.135:0/545783659 >> v1:172.21.15.194:6789/0 conn(0x7f6d780a9d70 legacy=0x7f6d7814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:31:46.232 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.237+0000 7f6d76ffd640 1 -- 172.21.15.135:0/545783659 >> v1:172.21.15.135:6789/0 conn(0x7f6d7813f650 legacy=0x7f6d780b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:31:46.234 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.238+0000 7f6d76ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:31:46.234 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.238+0000 7f6d76ffd640 1 monclient: found mon.noname-b 2024-03-29T11:31:46.234 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.238+0000 7f6d76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:46.234 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.238+0000 7f6d76ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:31:46.234 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.238+0000 7f6d76ffd640 1 -- 172.21.15.135:0/545783659 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d78146340 con 0x7f6d781457f0 2024-03-29T11:31:46.234 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.238+0000 7f6d76ffd640 10 monclient: _finish_auth 0 2024-03-29T11:31:46.235 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.238+0000 7f6d76ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:31:46.235 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.238+0000 7f6d76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:46.235 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.238+0000 7f6d76ffd640 1 -- 172.21.15.135:0/545783659 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f6d6c001690 con 0x7f6d781457f0 2024-03-29T11:31:46.235 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.238+0000 7f6d76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:46.235 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.238+0000 7f6d76ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:46.235 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.238+0000 7f6d76ffd640 10 monclient: dump: 2024-03-29T11:31:46.235 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:46.235 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:46.235 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:46.235 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:46.236 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:46.236 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:46.236 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:46.236 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:46.236 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:46.236 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:46.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.238+0000 7f6d7e365640 5 monclient: authenticate success, global_id 4499 2024-03-29T11:31:46.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.238+0000 7f6d7e365640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:46.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.238+0000 7f6d76ffd640 1 -- 172.21.15.135:0/545783659 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d6c001860 con 0x7f6d781457f0 2024-03-29T11:31:46.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.238+0000 7f6d76ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:46.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.238+0000 7f6d76ffd640 1 -- 172.21.15.135:0/545783659 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f6d6c001cb0 con 0x7f6d781457f0 2024-03-29T11:31:46.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.238+0000 7f6d76ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:46.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.238+0000 7f6d76ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:46.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.238+0000 7f6d76ffd640 10 monclient: dump: 2024-03-29T11:31:46.236 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:46.237 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:46.237 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:46.237 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:46.237 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:46.237 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:46.237 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:46.237 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:46.237 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:46.237 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:46.237 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.238+0000 7f6d7e365640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:46.237 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.238+0000 7f6d7e365640 4 set_mon_vals no callback set 2024-03-29T11:31:46.237 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.239+0000 7f6d7e365640 10 monclient: shutdown 2024-03-29T11:31:46.237 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.239+0000 7f6d7e365640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:46.237 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.239+0000 7f6d7e365640 1 -- 172.21.15.135:0/545783659 >> v1:172.21.15.135:6790/0 conn(0x7f6d781457f0 legacy=0x7f6d7813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:46.237 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.239+0000 7f6d7e365640 1 -- 172.21.15.135:0/545783659 shutdown_connections 2024-03-29T11:31:46.238 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.239+0000 7f6d7e365640 1 -- 172.21.15.135:0/545783659 >> 172.21.15.135:0/545783659 conn(0x7f6d78066a00 msgr2=0x7f6d78066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:46.238 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.242+0000 7f6d7e365640 1 -- 172.21.15.135:0/545783659 shutdown_connections 2024-03-29T11:31:46.238 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.243+0000 7f6d7e365640 1 -- 172.21.15.135:0/545783659 wait complete. 2024-03-29T11:31:46.239 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.244+0000 7f6d7e365640 10 monclient: build_initial_monmap 2024-03-29T11:31:46.239 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.244+0000 7f6d7e365640 10 monclient: monmap: 2024-03-29T11:31:46.239 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:31:46.239 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:46.239 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:31:46.239 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:31:46.239 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:46.239 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:46.239 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:31:46.239 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:31:46.239 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:31:46.239 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:46.239 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.245+0000 7f6d7e365640 1 librados: starting msgr at 2024-03-29T11:31:46.239 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.245+0000 7f6d7e365640 1 librados: starting objecter 2024-03-29T11:31:46.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.245+0000 7f6d7e365640 1 Processor -- start 2024-03-29T11:31:46.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.246+0000 7f6d7e365640 1 -- start start 2024-03-29T11:31:46.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.246+0000 7f6d7e365640 1 librados: setting wanted keys 2024-03-29T11:31:46.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.246+0000 7f6d7e365640 1 librados: calling monclient init 2024-03-29T11:31:46.241 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.246+0000 7f6d7e365640 10 monclient: init 2024-03-29T11:31:46.241 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.247+0000 7f6d7e365640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:46.241 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.247+0000 7f6d7e365640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:31:46.242 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.247+0000 7f6d7e365640 10 monclient(hunting): picked mon.noname-c con 0x7f6d780a9d70 addr v1:172.21.15.135:6790/0 2024-03-29T11:31:46.242 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.247+0000 7f6d7e365640 10 monclient(hunting): picked mon.noname-a con 0x7f6d7813f650 addr v1:172.21.15.135:6789/0 2024-03-29T11:31:46.242 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.247+0000 7f6d77fff640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f6d780a9d70 0x7f6d781398f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:56170/0 (socket says 172.21.15.135:56170) 2024-03-29T11:31:46.242 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.247+0000 7f6d7e365640 10 monclient(hunting): picked mon.noname-b con 0x7f6d781457f0 addr v1:172.21.15.194:6789/0 2024-03-29T11:31:46.242 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.247+0000 7f6d77fff640 1 -- 172.21.15.135:0/2500961471 learned_addr learned my addr 172.21.15.135:0/2500961471 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:31:46.242 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.247+0000 7f6d7e365640 1 -- 172.21.15.135:0/2500961471 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d78144b10 con 0x7f6d7813f650 2024-03-29T11:31:46.242 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.248+0000 7f6d74ff9640 10 client.?.objecter ms_handle_connect 0x7f6d780a9d70 2024-03-29T11:31:46.242 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.248+0000 7f6d74ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:46.246 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.251+0000 7f6d7e365640 1 -- 172.21.15.135:0/2500961471 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d78144fa0 con 0x7f6d781457f0 2024-03-29T11:31:46.246 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.251+0000 7f6d7e365640 1 -- 172.21.15.135:0/2500961471 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d78132740 con 0x7f6d780a9d70 2024-03-29T11:31:46.246 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.251+0000 7f6d7e365640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:46.246 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.251+0000 7f6d74ff9640 10 client.?.objecter ms_handle_connect 0x7f6d781457f0 2024-03-29T11:31:46.246 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.251+0000 7f6d74ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:46.246 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.251+0000 7f6d74ff9640 1 -- 172.21.15.135:0/2500961471 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158136491 0 0) 0x7f6d78132740 con 0x7f6d780a9d70 2024-03-29T11:31:46.246 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.251+0000 7f6d7e365640 10 monclient(hunting): authenticate will time out at 1213.619399s 2024-03-29T11:31:46.246 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.251+0000 7f6d74ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:31:46.246 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.251+0000 7f6d74ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:46.246 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.251+0000 7f6d74ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:46.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.251+0000 7f6d74ff9640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:31:46.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.251+0000 7f6d74ff9640 1 -- 172.21.15.135:0/2500961471 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d64004eb0 con 0x7f6d780a9d70 2024-03-29T11:31:46.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.251+0000 7f6d74ff9640 1 -- 172.21.15.135:0/2500961471 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029087260 0 0) 0x7f6d78144fa0 con 0x7f6d781457f0 2024-03-29T11:31:46.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.251+0000 7f6d74ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:31:46.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.251+0000 7f6d74ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:46.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.251+0000 7f6d74ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:46.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.251+0000 7f6d74ff9640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:31:46.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.251+0000 7f6d74ff9640 1 -- 172.21.15.135:0/2500961471 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d78132740 con 0x7f6d781457f0 2024-03-29T11:31:46.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.252+0000 7f6d74ff9640 10 client.?.objecter ms_handle_connect 0x7f6d7813f650 2024-03-29T11:31:46.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.252+0000 7f6d74ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:46.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.252+0000 7f6d74ff9640 1 -- 172.21.15.135:0/2500961471 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 456955651 0 0) 0x7f6d78144b10 con 0x7f6d7813f650 2024-03-29T11:31:46.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.252+0000 7f6d74ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:31:46.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.252+0000 7f6d74ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:46.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.252+0000 7f6d74ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:46.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.252+0000 7f6d74ff9640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:31:46.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.252+0000 7f6d74ff9640 1 -- 172.21.15.135:0/2500961471 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d78144fa0 con 0x7f6d7813f650 2024-03-29T11:31:46.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.252+0000 7f6d74ff9640 1 -- 172.21.15.135:0/2500961471 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 817469597 0 0) 0x7f6d64004eb0 con 0x7f6d780a9d70 2024-03-29T11:31:46.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.252+0000 7f6d74ff9640 1 -- 172.21.15.135:0/2500961471 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d78144b10 con 0x7f6d780a9d70 2024-03-29T11:31:46.249 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.252+0000 7f6d74ff9640 1 -- 172.21.15.135:0/2500961471 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f6d6c001c30 con 0x7f6d780a9d70 2024-03-29T11:31:46.249 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.252+0000 7f6d74ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:46.249 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.252+0000 7f6d74ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:46.249 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.252+0000 7f6d74ff9640 10 monclient(hunting): dump: 2024-03-29T11:31:46.249 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:46.249 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:46.249 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:46.249 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:46.249 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:46.249 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:46.249 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:46.250 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:46.250 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:46.250 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:46.250 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.252+0000 7f6d74ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:46.250 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.252+0000 7f6d74ff9640 1 -- 172.21.15.135:0/2500961471 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 93163336 0 0) 0x7f6d78144b10 con 0x7f6d780a9d70 2024-03-29T11:31:46.250 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.252+0000 7f6d74ff9640 1 -- 172.21.15.135:0/2500961471 >> v1:172.21.15.194:6789/0 conn(0x7f6d781457f0 legacy=0x7f6d78132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:46.250 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.252+0000 7f6d74ff9640 1 -- 172.21.15.135:0/2500961471 >> v1:172.21.15.135:6789/0 conn(0x7f6d7813f650 legacy=0x7f6d78131920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:46.250 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.252+0000 7f6d74ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:31:46.250 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.252+0000 7f6d74ff9640 1 monclient: found mon.c 2024-03-29T11:31:46.250 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.252+0000 7f6d74ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:46.250 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.252+0000 7f6d74ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:46.250 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.252+0000 7f6d74ff9640 1 -- 172.21.15.135:0/2500961471 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d78138300 con 0x7f6d780a9d70 2024-03-29T11:31:46.250 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.253+0000 7f6d74ff9640 10 monclient: _finish_auth 0 2024-03-29T11:31:46.250 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.253+0000 7f6d74ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:31:46.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.253+0000 7f6d74ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:46.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.253+0000 7f6d7e365640 5 monclient: authenticate success, global_id 4502 2024-03-29T11:31:46.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.253+0000 7f6d7e365640 10 monclient: _renew_subs 2024-03-29T11:31:46.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.253+0000 7f6d7e365640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:46.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.253+0000 7f6d7e365640 1 -- 172.21.15.135:0/2500961471 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d78139180 con 0x7f6d780a9d70 2024-03-29T11:31:46.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.253+0000 7f6d7e365640 10 client.4502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:46.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.253+0000 7f6d7e365640 10 monclient: _renew_subs 2024-03-29T11:31:46.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.253+0000 7f6d7e365640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:46.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.253+0000 7f6d7e365640 1 -- 172.21.15.135:0/2500961471 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d78144800 con 0x7f6d780a9d70 2024-03-29T11:31:46.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.253+0000 7f6d7e365640 1 librados: init done 2024-03-29T11:31:46.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.253+0000 7f6d74ff9640 1 -- 172.21.15.135:0/2500961471 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d6c004a90 con 0x7f6d780a9d70 2024-03-29T11:31:46.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.253+0000 7f6d74ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:46.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.253+0000 7f6d74ff9640 1 -- 172.21.15.135:0/2500961471 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f6d6c004e20 con 0x7f6d780a9d70 2024-03-29T11:31:46.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.253+0000 7f6d74ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:46.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.253+0000 7f6d74ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:46.252 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.253+0000 7f6d74ff9640 10 monclient: dump: 2024-03-29T11:31:46.252 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:46.252 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:46.252 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:46.252 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:46.252 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:46.252 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:46.252 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:46.252 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:46.252 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:46.252 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:46.252 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.253+0000 7f6d75ffb640 4 set_mon_vals no callback set 2024-03-29T11:31:46.252 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.254+0000 7f6d7e365640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:46.252 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.254+0000 7f6d7e365640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:46.252 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.254+0000 7f6d7e365640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:46.253 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.254+0000 7f6d7e365640 1 -- 172.21.15.135:0/2500961471 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d58005080 con 0x7f6d780a9d70 2024-03-29T11:31:46.253 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.256+0000 7f6d74ff9640 1 -- 172.21.15.135:0/2500961471 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f6d6c004f80 con 0x7f6d780a9d70 2024-03-29T11:31:46.253 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.256+0000 7f6d74ff9640 1 -- 172.21.15.135:0/2500961471 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f6d6c084b00 con 0x7f6d780a9d70 2024-03-29T11:31:46.253 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.256+0000 7f6d74ff9640 10 client.4502.objecter ms_dispatch 0x7f6d78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:31:46.253 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.256+0000 7f6d74ff9640 3 client.4502.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:31:46.253 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.256+0000 7f6d74ff9640 3 client.4502.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:31:46.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.257+0000 7f6d74ff9640 20 client.4502.objecter dump_active .. 0 homeless 2024-03-29T11:31:46.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.259+0000 7f6d777fe640 10 monclient: get_auth_request con 0x7f6d6406de40 auth_method 0 2024-03-29T11:31:46.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.259+0000 7f6d74ff9640 10 client.4502.objecter ms_handle_connect 0x7f6d6406de40 2024-03-29T11:31:46.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.265+0000 7f6d74ff9640 1 -- 172.21.15.135:0/2500961471 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6d6c0503e0 con 0x7f6d780a9d70 2024-03-29T11:31:46.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.265+0000 7f6d74ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:46.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.265+0000 7f6d74ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:46.579 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.585+0000 7f6d7e365640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:46.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.585+0000 7f6d7e365640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:46.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.585+0000 7f6d7e365640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:46.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.585+0000 7f6d7e365640 1 -- 172.21.15.135:0/2500961471 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d58005330 con 0x7f6d780a9d70 2024-03-29T11:31:46.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.585+0000 7f6d74ff9640 1 -- 172.21.15.135:0/2500961471 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f6d6c052310 con 0x7f6d780a9d70 2024-03-29T11:31:46.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.585+0000 7f6d74ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:46.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.585+0000 7f6d74ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:46.581 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:31:46.581 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:46.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.589+0000 7f6d7e365640 10 librados: watch_flush enter 2024-03-29T11:31:46.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.589+0000 7f6d7e365640 10 librados: watch_flush exit 2024-03-29T11:31:46.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.589+0000 7f6d7e365640 20 client.4502.objecter shutdown clearing up homeless session... 2024-03-29T11:31:46.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.589+0000 7f6d7e365640 10 client.4502.objecter successfully canceled tick 2024-03-29T11:31:46.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.590+0000 7f6d7e365640 1 -- 172.21.15.135:0/2500961471 >> v1:172.21.15.135:6816/3469875285 conn(0x7f6d6406de40 legacy=0x7f6d64070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:46.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.590+0000 7f6d7e365640 10 monclient: shutdown 2024-03-29T11:31:46.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.590+0000 7f6d7e365640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:46.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.590+0000 7f6d7e365640 1 -- 172.21.15.135:0/2500961471 >> v1:172.21.15.135:6790/0 conn(0x7f6d780a9d70 legacy=0x7f6d781398f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:46.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.590+0000 7f6d7e365640 1 -- 172.21.15.135:0/2500961471 shutdown_connections 2024-03-29T11:31:46.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.590+0000 7f6d7e365640 1 -- 172.21.15.135:0/2500961471 >> 172.21.15.135:0/2500961471 conn(0x7f6d78066a00 msgr2=0x7f6d7813a610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:46.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.590+0000 7f6d7e365640 1 -- 172.21.15.135:0/2500961471 shutdown_connections 2024-03-29T11:31:46.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.591+0000 7f6d7e365640 1 -- 172.21.15.135:0/2500961471 wait complete. 2024-03-29T11:31:46.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:46.591+0000 7f6d7e365640 1 librados: shutdown 2024-03-29T11:31:46.602 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:53.410 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:53.553 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.558+0000 7fa0c8ec5640 10 monclient: get_monmap_and_config 2024-03-29T11:31:53.553 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.558+0000 7fa0c8ec5640 10 monclient: build_initial_monmap 2024-03-29T11:31:53.553 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.558+0000 7fa0c8ec5640 10 monclient: monmap: 2024-03-29T11:31:53.553 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:31:53.553 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:53.553 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:31:53.559228+0000 2024-03-29T11:31:53.554 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:31:53.559228+0000 2024-03-29T11:31:53.554 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:53.554 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:53.554 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:31:53.554 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:31:53.554 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:31:53.554 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:53.557 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.559+0000 7fa0c8ec5640 1 Processor -- start 2024-03-29T11:31:53.557 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.559+0000 7fa0c8ec5640 1 -- start start 2024-03-29T11:31:53.557 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.559+0000 7fa0c8ec5640 10 monclient: init 2024-03-29T11:31:53.557 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.560+0000 7fa0c8ec5640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:53.557 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.560+0000 7fa0c8ec5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:31:53.557 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.561+0000 7fa0c8ec5640 10 monclient(hunting): picked mon.noname-a con 0x7fa0c40b0860 addr v1:172.21.15.135:6789/0 2024-03-29T11:31:53.557 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.561+0000 7fa0c1d9b640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7fa0c413ef30 0x7fa0c40af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:39560/0 (socket says 172.21.15.135:39560) 2024-03-29T11:31:53.558 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.561+0000 7fa0c8ec5640 10 monclient(hunting): picked mon.noname-b con 0x7fa0c413ef30 addr v1:172.21.15.135:6790/0 2024-03-29T11:31:53.558 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.561+0000 7fa0c1d9b640 1 -- 172.21.15.135:0/3298462777 learned_addr learned my addr 172.21.15.135:0/3298462777 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:31:53.558 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.561+0000 7fa0c8ec5640 10 monclient(hunting): picked mon.noname-c con 0x7fa0c40af8e0 addr v1:172.21.15.194:6789/0 2024-03-29T11:31:53.558 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.561+0000 7fa0c8ec5640 1 -- 172.21.15.135:0/3298462777 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c40b0c40 con 0x7fa0c40b0860 2024-03-29T11:31:53.558 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.561+0000 7fa0c8ec5640 1 -- 172.21.15.135:0/3298462777 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c40aed10 con 0x7fa0c40af8e0 2024-03-29T11:31:53.558 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.561+0000 7fa0c8ec5640 1 -- 172.21.15.135:0/3298462777 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c40afd50 con 0x7fa0c413ef30 2024-03-29T11:31:53.558 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.561+0000 7fa0c8ec5640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:53.558 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.561+0000 7fa0c8ec5640 10 monclient(hunting): authenticate will time out at 1220.929513s 2024-03-29T11:31:53.558 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.561+0000 7fa0c159a640 1 -- 172.21.15.135:0/3298462777 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3400581215 0 0) 0x7fa0c40afd50 con 0x7fa0c413ef30 2024-03-29T11:31:53.559 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.561+0000 7fa0c159a640 10 monclient(hunting): _negotiate 2024-03-29T11:31:53.559 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.561+0000 7fa0c159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:53.559 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.561+0000 7fa0c159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:53.559 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.561+0000 7fa0c159a640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:31:53.559 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.561+0000 7fa0c159a640 1 -- 172.21.15.135:0/3298462777 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0ac003520 con 0x7fa0c413ef30 2024-03-29T11:31:53.559 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.561+0000 7fa0c159a640 1 -- 172.21.15.135:0/3298462777 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4112780835 0 0) 0x7fa0ac003520 con 0x7fa0c413ef30 2024-03-29T11:31:53.559 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.561+0000 7fa0c159a640 1 -- 172.21.15.135:0/3298462777 >> v1:172.21.15.194:6789/0 conn(0x7fa0c40af8e0 legacy=0x7fa0c4149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:53.559 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.561+0000 7fa0c159a640 1 -- 172.21.15.135:0/3298462777 >> v1:172.21.15.135:6789/0 conn(0x7fa0c40b0860 legacy=0x7fa0c413e820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:31:53.570 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.564+0000 7fa0c159a640 10 monclient: _finish_hunting 0 2024-03-29T11:31:53.571 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.564+0000 7fa0c159a640 1 monclient: found mon.noname-b 2024-03-29T11:31:53.571 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.564+0000 7fa0c159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:53.571 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.564+0000 7fa0c159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:31:53.571 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.564+0000 7fa0c159a640 1 -- 172.21.15.135:0/3298462777 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c40ab1b0 con 0x7fa0c413ef30 2024-03-29T11:31:53.571 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.564+0000 7fa0c159a640 10 monclient: _finish_auth 0 2024-03-29T11:31:53.571 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.564+0000 7fa0c159a640 10 monclient: _check_auth_tickets 2024-03-29T11:31:53.571 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.564+0000 7fa0c159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:53.571 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.564+0000 7fa0c159a640 1 -- 172.21.15.135:0/3298462777 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fa0b8001690 con 0x7fa0c413ef30 2024-03-29T11:31:53.571 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.565+0000 7fa0c8ec5640 5 monclient: authenticate success, global_id 4505 2024-03-29T11:31:53.571 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.565+0000 7fa0c8ec5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:53.572 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.565+0000 7fa0c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:53.572 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.565+0000 7fa0c159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:53.572 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.565+0000 7fa0c159a640 10 monclient: dump: 2024-03-29T11:31:53.572 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:53.572 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:53.572 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:53.572 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:53.572 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:53.573 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:53.573 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:53.573 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:53.573 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:53.573 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:53.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.565+0000 7fa0c8ec5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:53.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.565+0000 7fa0c159a640 1 -- 172.21.15.135:0/3298462777 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0b8001920 con 0x7fa0c413ef30 2024-03-29T11:31:53.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.565+0000 7fa0c159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:53.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.565+0000 7fa0c8ec5640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:53.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.565+0000 7fa0c8ec5640 4 set_mon_vals no callback set 2024-03-29T11:31:53.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.565+0000 7fa0c159a640 1 -- 172.21.15.135:0/3298462777 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fa0b8001cf0 con 0x7fa0c413ef30 2024-03-29T11:31:53.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.565+0000 7fa0c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:53.574 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.565+0000 7fa0c159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:53.574 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.565+0000 7fa0c159a640 10 monclient: dump: 2024-03-29T11:31:53.574 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:53.574 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:53.574 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:53.574 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:53.574 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:53.574 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:53.574 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:53.574 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:53.574 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:53.575 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:53.575 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.566+0000 7fa0c8ec5640 10 monclient: shutdown 2024-03-29T11:31:53.575 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.566+0000 7fa0c8ec5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:53.575 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.566+0000 7fa0c8ec5640 1 -- 172.21.15.135:0/3298462777 >> v1:172.21.15.135:6790/0 conn(0x7fa0c413ef30 legacy=0x7fa0c40af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:53.575 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.566+0000 7fa0c8ec5640 1 -- 172.21.15.135:0/3298462777 shutdown_connections 2024-03-29T11:31:53.575 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.566+0000 7fa0c8ec5640 1 -- 172.21.15.135:0/3298462777 >> 172.21.15.135:0/3298462777 conn(0x7fa0c406b660 msgr2=0x7fa0c406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:53.575 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.567+0000 7fa0c8ec5640 1 -- 172.21.15.135:0/3298462777 shutdown_connections 2024-03-29T11:31:53.575 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.567+0000 7fa0c8ec5640 1 -- 172.21.15.135:0/3298462777 wait complete. 2024-03-29T11:31:53.575 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.568+0000 7fa0c8ec5640 10 monclient: build_initial_monmap 2024-03-29T11:31:53.575 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.568+0000 7fa0c8ec5640 10 monclient: monmap: 2024-03-29T11:31:53.575 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:31:53.576 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:53.576 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:31:53.576 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:31:53.576 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:53.576 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:53.576 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:31:53.576 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:31:53.576 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:31:53.576 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:53.576 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.569+0000 7fa0c8ec5640 1 librados: starting msgr at 2024-03-29T11:31:53.576 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.569+0000 7fa0c8ec5640 1 librados: starting objecter 2024-03-29T11:31:53.577 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.569+0000 7fa0c8ec5640 1 Processor -- start 2024-03-29T11:31:53.577 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.569+0000 7fa0c8ec5640 1 -- start start 2024-03-29T11:31:53.577 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.569+0000 7fa0c8ec5640 1 librados: setting wanted keys 2024-03-29T11:31:53.577 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.569+0000 7fa0c8ec5640 1 librados: calling monclient init 2024-03-29T11:31:53.577 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.569+0000 7fa0c8ec5640 10 monclient: init 2024-03-29T11:31:53.577 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.570+0000 7fa0c8ec5640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:53.577 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.570+0000 7fa0c8ec5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:31:53.577 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.570+0000 7fa0c8ec5640 10 monclient(hunting): picked mon.noname-b con 0x7fa0c40af8e0 addr v1:172.21.15.194:6789/0 2024-03-29T11:31:53.577 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.570+0000 7fa0c8ec5640 10 monclient(hunting): picked mon.noname-a con 0x7fa0c40b0860 addr v1:172.21.15.135:6789/0 2024-03-29T11:31:53.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.570+0000 7fa0c8ec5640 10 monclient(hunting): picked mon.noname-c con 0x7fa0c413ef30 addr v1:172.21.15.135:6790/0 2024-03-29T11:31:53.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.570+0000 7fa0c8ec5640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c40b3840 con 0x7fa0c40b0860 2024-03-29T11:31:53.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.570+0000 7fa0c8ec5640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c41d90a0 con 0x7fa0c40af8e0 2024-03-29T11:31:53.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.570+0000 7fa0c8ec5640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c414c780 con 0x7fa0c413ef30 2024-03-29T11:31:53.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.570+0000 7fa0c8ec5640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:53.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.570+0000 7fa0c8ec5640 10 monclient(hunting): authenticate will time out at 1220.938765s 2024-03-29T11:31:53.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.570+0000 7fa0c259c640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fa0c40af8e0 0x7fa0c41d8990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:42162/0 (socket says 172.21.15.135:42162) 2024-03-29T11:31:53.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.570+0000 7fa0c259c640 1 -- 172.21.15.135:0/2306766339 learned_addr learned my addr 172.21.15.135:0/2306766339 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:31:53.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.571+0000 7fa0b2ffd640 10 client.?.objecter ms_handle_connect 0x7fa0c40af8e0 2024-03-29T11:31:53.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.571+0000 7fa0b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:53.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.571+0000 7fa0b2ffd640 10 client.?.objecter ms_handle_connect 0x7fa0c413ef30 2024-03-29T11:31:53.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.571+0000 7fa0b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:53.579 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.571+0000 7fa0b2ffd640 1 -- 172.21.15.135:0/2306766339 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401584700 0 0) 0x7fa0c414c780 con 0x7fa0c413ef30 2024-03-29T11:31:53.579 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.571+0000 7fa0b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:53.579 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.571+0000 7fa0b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:53.579 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.571+0000 7fa0b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:53.579 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.571+0000 7fa0b2ffd640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:31:53.579 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.571+0000 7fa0b2ffd640 1 -- 172.21.15.135:0/2306766339 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0a4004eb0 con 0x7fa0c413ef30 2024-03-29T11:31:53.579 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.571+0000 7fa0b2ffd640 1 -- 172.21.15.135:0/2306766339 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 461989622 0 0) 0x7fa0c41d90a0 con 0x7fa0c40af8e0 2024-03-29T11:31:53.579 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.571+0000 7fa0b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:53.579 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.571+0000 7fa0b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:53.579 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.571+0000 7fa0b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:53.579 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.571+0000 7fa0b2ffd640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:31:53.579 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.571+0000 7fa0b2ffd640 1 -- 172.21.15.135:0/2306766339 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0c414c780 con 0x7fa0c40af8e0 2024-03-29T11:31:53.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.572+0000 7fa0b2ffd640 1 -- 172.21.15.135:0/2306766339 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3194401899 0 0) 0x7fa0a4004eb0 con 0x7fa0c413ef30 2024-03-29T11:31:53.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.572+0000 7fa0b2ffd640 1 -- 172.21.15.135:0/2306766339 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0c41d90a0 con 0x7fa0c413ef30 2024-03-29T11:31:53.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.572+0000 7fa0b2ffd640 1 -- 172.21.15.135:0/2306766339 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fa0bc001890 con 0x7fa0c413ef30 2024-03-29T11:31:53.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.572+0000 7fa0b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:53.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.572+0000 7fa0b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:53.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.572+0000 7fa0b2ffd640 10 monclient(hunting): dump: 2024-03-29T11:31:53.580 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:53.580 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:53.580 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:53.580 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:53.580 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:53.580 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:53.581 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:53.581 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:53.581 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:53.581 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:53.581 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.572+0000 7fa0b2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:53.581 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.572+0000 7fa0b2ffd640 1 -- 172.21.15.135:0/2306766339 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1672163975 0 0) 0x7fa0c414c780 con 0x7fa0c40af8e0 2024-03-29T11:31:53.581 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.572+0000 7fa0b2ffd640 1 -- 172.21.15.135:0/2306766339 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0a4004eb0 con 0x7fa0c40af8e0 2024-03-29T11:31:53.582 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.572+0000 7fa0b2ffd640 1 -- 172.21.15.135:0/2306766339 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fa0b40018a0 con 0x7fa0c40af8e0 2024-03-29T11:31:53.582 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.572+0000 7fa0b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:53.582 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.572+0000 7fa0b2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:53.582 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.572+0000 7fa0b2ffd640 10 monclient(hunting): dump: 2024-03-29T11:31:53.582 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:53.582 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:53.582 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:53.582 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:53.582 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:53.582 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:53.582 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:53.583 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:53.583 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:53.583 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:53.583 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.572+0000 7fa0b2ffd640 1 -- 172.21.15.135:0/2306766339 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4059037514 0 0) 0x7fa0c41d90a0 con 0x7fa0c413ef30 2024-03-29T11:31:53.583 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.572+0000 7fa0b2ffd640 1 -- 172.21.15.135:0/2306766339 >> v1:172.21.15.194:6789/0 conn(0x7fa0c40af8e0 legacy=0x7fa0c41d8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:53.583 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.572+0000 7fa0b2ffd640 1 -- 172.21.15.135:0/2306766339 >> v1:172.21.15.135:6789/0 conn(0x7fa0c40b0860 legacy=0x7fa0c414b960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:53.583 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.574+0000 7fa0b2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:31:53.583 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.574+0000 7fa0b2ffd640 1 monclient: found mon.c 2024-03-29T11:31:53.583 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.574+0000 7fa0b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:53.583 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.574+0000 7fa0b2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:53.583 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.574+0000 7fa0b2ffd640 1 -- 172.21.15.135:0/2306766339 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c406e410 con 0x7fa0c413ef30 2024-03-29T11:31:53.583 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.574+0000 7fa0b2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:31:53.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.574+0000 7fa0b2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:31:53.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.574+0000 7fa0b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:53.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.574+0000 7fa0c8ec5640 5 monclient: authenticate success, global_id 4508 2024-03-29T11:31:53.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.574+0000 7fa0c8ec5640 10 monclient: _renew_subs 2024-03-29T11:31:53.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.574+0000 7fa0c8ec5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:53.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.574+0000 7fa0c8ec5640 1 -- 172.21.15.135:0/2306766339 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c406e6f0 con 0x7fa0c413ef30 2024-03-29T11:31:53.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.574+0000 7fa0c8ec5640 10 client.4508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:53.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.574+0000 7fa0c8ec5640 10 monclient: _renew_subs 2024-03-29T11:31:53.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.574+0000 7fa0c8ec5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:53.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.574+0000 7fa0c8ec5640 1 -- 172.21.15.135:0/2306766339 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c414cef0 con 0x7fa0c413ef30 2024-03-29T11:31:53.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.574+0000 7fa0c8ec5640 1 librados: init done 2024-03-29T11:31:53.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.575+0000 7fa0b0ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:53.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.575+0000 7fa0b0ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:53.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.575+0000 7fa0b0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:53.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.575+0000 7fa0b0ff9640 1 -- 172.21.15.135:0/2306766339 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0a8005080 con 0x7fa0c413ef30 2024-03-29T11:31:53.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.576+0000 7fa0b2ffd640 1 -- 172.21.15.135:0/2306766339 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0bc001da0 con 0x7fa0c413ef30 2024-03-29T11:31:53.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.576+0000 7fa0b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:53.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.576+0000 7fa0b2ffd640 1 -- 172.21.15.135:0/2306766339 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fa0bc003c10 con 0x7fa0c413ef30 2024-03-29T11:31:53.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.576+0000 7fa0b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:53.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.576+0000 7fa0b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:53.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.576+0000 7fa0b2ffd640 10 monclient: dump: 2024-03-29T11:31:53.585 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:31:53.585 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:31:53.585 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:53.585 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:31:53.586 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:53.586 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:31:53.586 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:31:53.586 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:31:53.586 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:31:53.586 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:31:53.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.577+0000 7fa0b2ffd640 1 -- 172.21.15.135:0/2306766339 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fa0bc019be0 con 0x7fa0c413ef30 2024-03-29T11:31:53.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.577+0000 7fa0c0d99640 4 set_mon_vals no callback set 2024-03-29T11:31:53.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.578+0000 7fa0b2ffd640 1 -- 172.21.15.135:0/2306766339 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7fa0bc083aa0 con 0x7fa0c413ef30 2024-03-29T11:31:53.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.578+0000 7fa0b2ffd640 10 client.4508.objecter ms_dispatch 0x7fa0c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:31:53.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.578+0000 7fa0b2ffd640 3 client.4508.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:31:53.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.578+0000 7fa0b2ffd640 3 client.4508.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:31:53.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.578+0000 7fa0b2ffd640 20 client.4508.objecter dump_active .. 0 homeless 2024-03-29T11:31:53.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.578+0000 7fa0c259c640 10 monclient: get_auth_request con 0x7fa0a40722b0 auth_method 0 2024-03-29T11:31:53.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.578+0000 7fa0b2ffd640 10 client.4508.objecter ms_handle_connect 0x7fa0a40722b0 2024-03-29T11:31:53.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.583+0000 7fa0b2ffd640 1 -- 172.21.15.135:0/2306766339 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa0bc04f380 con 0x7fa0c413ef30 2024-03-29T11:31:53.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.583+0000 7fa0b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:53.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.583+0000 7fa0b2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:53.879 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.885+0000 7fa0b0ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:53.879 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.885+0000 7fa0b0ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:53.879 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.885+0000 7fa0b0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:31:53.879 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.885+0000 7fa0b0ff9640 1 -- 172.21.15.135:0/2306766339 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0a8005330 con 0x7fa0c413ef30 2024-03-29T11:31:53.880 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.886+0000 7fa0b2ffd640 1 -- 172.21.15.135:0/2306766339 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7fa0bc0512b0 con 0x7fa0c413ef30 2024-03-29T11:31:53.880 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.886+0000 7fa0b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:53.880 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.886+0000 7fa0b2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:53.880 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:31:53.880 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:53.883 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.888+0000 7fa0b0ff9640 10 librados: watch_flush enter 2024-03-29T11:31:53.883 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.888+0000 7fa0b0ff9640 10 librados: watch_flush exit 2024-03-29T11:31:53.883 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.888+0000 7fa0b0ff9640 20 client.4508.objecter shutdown clearing up homeless session... 2024-03-29T11:31:53.883 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.888+0000 7fa0b0ff9640 10 client.4508.objecter successfully canceled tick 2024-03-29T11:31:53.883 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.889+0000 7fa0b0ff9640 1 -- 172.21.15.135:0/2306766339 >> v1:172.21.15.135:6816/3469875285 conn(0x7fa0a40722b0 legacy=0x7fa0a4074770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:53.883 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.889+0000 7fa0b0ff9640 10 monclient: shutdown 2024-03-29T11:31:53.884 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.889+0000 7fa0b0ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:53.884 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.889+0000 7fa0b0ff9640 1 -- 172.21.15.135:0/2306766339 >> v1:172.21.15.135:6790/0 conn(0x7fa0c413ef30 legacy=0x7fa0c414c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:53.884 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.889+0000 7fa0b0ff9640 1 -- 172.21.15.135:0/2306766339 shutdown_connections 2024-03-29T11:31:53.884 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.889+0000 7fa0b0ff9640 1 -- 172.21.15.135:0/2306766339 >> 172.21.15.135:0/2306766339 conn(0x7fa0c406b660 msgr2=0x7fa0c413dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:53.884 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.889+0000 7fa0b0ff9640 1 -- 172.21.15.135:0/2306766339 shutdown_connections 2024-03-29T11:31:53.884 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.889+0000 7fa0b0ff9640 1 -- 172.21.15.135:0/2306766339 wait complete. 2024-03-29T11:31:53.884 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:31:53.889+0000 7fa0b0ff9640 1 librados: shutdown 2024-03-29T11:31:53.898 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:00.707 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:00.855 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.859+0000 7f01b02c0640 10 monclient: get_monmap_and_config 2024-03-29T11:32:00.856 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.859+0000 7f01b02c0640 10 monclient: build_initial_monmap 2024-03-29T11:32:00.856 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.859+0000 7f01b02c0640 10 monclient: monmap: 2024-03-29T11:32:00.856 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:32:00.856 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:00.856 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:32:00.861141+0000 2024-03-29T11:32:00.856 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:32:00.861141+0000 2024-03-29T11:32:00.856 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:00.856 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:00.856 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:32:00.856 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:32:00.856 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:32:00.856 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:00.857 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.860+0000 7f01b02c0640 1 Processor -- start 2024-03-29T11:32:00.857 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.860+0000 7f01b02c0640 1 -- start start 2024-03-29T11:32:00.857 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.860+0000 7f01b02c0640 10 monclient: init 2024-03-29T11:32:00.859 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.861+0000 7f01b02c0640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:00.859 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.861+0000 7f01b02c0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:32:00.859 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.861+0000 7f01b02c0640 10 monclient(hunting): picked mon.noname-a con 0x7f01a80a9ac0 addr v1:172.21.15.135:6789/0 2024-03-29T11:32:00.859 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.861+0000 7f01b02c0640 10 monclient(hunting): picked mon.noname-b con 0x7f01a813ee90 addr v1:172.21.15.135:6790/0 2024-03-29T11:32:00.859 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.861+0000 7f01b02c0640 10 monclient(hunting): picked mon.noname-c con 0x7f01a80af900 addr v1:172.21.15.194:6789/0 2024-03-29T11:32:00.860 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.861+0000 7f01b02c0640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a80afd10 con 0x7f01a80a9ac0 2024-03-29T11:32:00.860 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.861+0000 7f01b02c0640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a80af6f0 con 0x7f01a80af900 2024-03-29T11:32:00.860 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.861+0000 7f01b02c0640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a80aac80 con 0x7f01a813ee90 2024-03-29T11:32:00.860 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.861+0000 7f01b02c0640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:00.860 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.861+0000 7f01b02c0640 10 monclient(hunting): authenticate will time out at 1228.230673s 2024-03-29T11:32:00.860 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.861+0000 7f01ad85b640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f01a813ee90 0x7f01a813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:34108/0 (socket says 172.21.15.135:34108) 2024-03-29T11:32:00.860 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.861+0000 7f01ad85b640 1 -- 172.21.15.135:0/3731806889 learned_addr learned my addr 172.21.15.135:0/3731806889 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:32:00.860 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.861+0000 7f01ae05c640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f01a80a9ac0 0x7f01a80adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:34474/0 (socket says 172.21.15.135:34474) 2024-03-29T11:32:00.860 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 1 -- 172.21.15.135:0/3731806889 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 505567862 0 0) 0x7f01a80aac80 con 0x7f01a813ee90 2024-03-29T11:32:00.860 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 10 monclient(hunting): _negotiate 2024-03-29T11:32:00.860 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:00.860 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:00.860 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:32:00.861 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 1 -- 172.21.15.135:0/3731806889 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0194003520 con 0x7f01a813ee90 2024-03-29T11:32:00.861 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 1 -- 172.21.15.135:0/3731806889 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 282083461 0 0) 0x7f01a80af6f0 con 0x7f01a80af900 2024-03-29T11:32:00.861 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 10 monclient(hunting): _negotiate 2024-03-29T11:32:00.861 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:00.861 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:00.861 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:32:00.861 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 1 -- 172.21.15.135:0/3731806889 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01a80aac80 con 0x7f01a80af900 2024-03-29T11:32:00.861 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 1 -- 172.21.15.135:0/3731806889 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 686742707 0 0) 0x7f01a80afd10 con 0x7f01a80a9ac0 2024-03-29T11:32:00.861 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 10 monclient(hunting): _negotiate 2024-03-29T11:32:00.861 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:00.861 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:00.861 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:32:00.861 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 1 -- 172.21.15.135:0/3731806889 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01940039d0 con 0x7f01a80a9ac0 2024-03-29T11:32:00.861 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 1 -- 172.21.15.135:0/3731806889 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 361198686 0 0) 0x7f01a80aac80 con 0x7f01a80af900 2024-03-29T11:32:00.862 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 1 -- 172.21.15.135:0/3731806889 >> v1:172.21.15.135:6790/0 conn(0x7f01a813ee90 legacy=0x7f01a813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:00.862 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 1 -- 172.21.15.135:0/3731806889 >> v1:172.21.15.135:6789/0 conn(0x7f01a80a9ac0 legacy=0x7f01a80adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:00.862 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 10 monclient: _finish_hunting 0 2024-03-29T11:32:00.862 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 1 monclient: found mon.noname-c 2024-03-29T11:32:00.862 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:00.862 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:32:00.862 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 1 -- 172.21.15.135:0/3731806889 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a80aaf60 con 0x7f01a80af900 2024-03-29T11:32:00.862 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 10 monclient: _finish_auth 0 2024-03-29T11:32:00.862 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 10 monclient: _check_auth_tickets 2024-03-29T11:32:00.862 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:00.862 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01b02c0640 5 monclient: authenticate success, global_id 4498 2024-03-29T11:32:00.862 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01b02c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:00.862 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 1 -- 172.21.15.135:0/3731806889 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f01980015f0 con 0x7f01a80af900 2024-03-29T11:32:00.862 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.862+0000 7f01ad05a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:00.863 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.863+0000 7f01ad05a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:00.863 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.863+0000 7f01ad05a640 10 monclient: dump: 2024-03-29T11:32:00.863 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:00.863 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:00.863 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:00.863 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:00.863 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:00.863 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:00.863 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:00.863 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:00.863 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:00.863 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:00.863 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.863+0000 7f01b02c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:00.864 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.863+0000 7f01ad05a640 1 -- 172.21.15.135:0/3731806889 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01980017d0 con 0x7f01a80af900 2024-03-29T11:32:00.864 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.863+0000 7f01ad05a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:00.864 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.863+0000 7f01b02c0640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:00.864 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.863+0000 7f01b02c0640 4 set_mon_vals no callback set 2024-03-29T11:32:00.864 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.863+0000 7f01ad05a640 1 -- 172.21.15.135:0/3731806889 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f0198001c20 con 0x7f01a80af900 2024-03-29T11:32:00.864 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.863+0000 7f01ad05a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:00.864 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.863+0000 7f01ad05a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:00.864 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.863+0000 7f01ad05a640 10 monclient: dump: 2024-03-29T11:32:00.864 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:00.864 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:00.864 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:00.864 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:00.864 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:00.864 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:00.865 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:00.865 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:00.865 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:00.865 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:00.865 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.863+0000 7f01b02c0640 10 monclient: shutdown 2024-03-29T11:32:00.865 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.863+0000 7f01b02c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:00.865 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.863+0000 7f01b02c0640 1 -- 172.21.15.135:0/3731806889 >> v1:172.21.15.194:6789/0 conn(0x7f01a80af900 legacy=0x7f01a8149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:00.865 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.864+0000 7f01b02c0640 1 -- 172.21.15.135:0/3731806889 shutdown_connections 2024-03-29T11:32:00.865 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.864+0000 7f01b02c0640 1 -- 172.21.15.135:0/3731806889 >> 172.21.15.135:0/3731806889 conn(0x7f01a80b35e0 msgr2=0x7f01a80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:00.865 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.864+0000 7f01b02c0640 1 -- 172.21.15.135:0/3731806889 shutdown_connections 2024-03-29T11:32:00.866 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.864+0000 7f01b02c0640 1 -- 172.21.15.135:0/3731806889 wait complete. 2024-03-29T11:32:00.871 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.865+0000 7f01b02c0640 10 monclient: build_initial_monmap 2024-03-29T11:32:00.871 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.865+0000 7f01b02c0640 10 monclient: monmap: 2024-03-29T11:32:00.871 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:32:00.872 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:00.872 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:32:00.872 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:32:00.872 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:00.872 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:00.872 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:32:00.872 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:32:00.872 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:32:00.872 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:00.872 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.865+0000 7f01b02c0640 1 librados: starting msgr at 2024-03-29T11:32:00.872 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.865+0000 7f01b02c0640 1 librados: starting objecter 2024-03-29T11:32:00.872 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.865+0000 7f01b02c0640 1 Processor -- start 2024-03-29T11:32:00.872 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.865+0000 7f01b02c0640 1 -- start start 2024-03-29T11:32:00.872 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.865+0000 7f01b02c0640 1 librados: setting wanted keys 2024-03-29T11:32:00.873 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.865+0000 7f01b02c0640 1 librados: calling monclient init 2024-03-29T11:32:00.873 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.865+0000 7f01b02c0640 10 monclient: init 2024-03-29T11:32:00.873 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.866+0000 7f01b02c0640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:00.873 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.866+0000 7f01b02c0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:32:00.873 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.866+0000 7f01b02c0640 10 monclient(hunting): picked mon.noname-b con 0x7f01a80a9ac0 addr v1:172.21.15.194:6789/0 2024-03-29T11:32:00.873 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.866+0000 7f01b02c0640 10 monclient(hunting): picked mon.noname-a con 0x7f01a80af900 addr v1:172.21.15.135:6789/0 2024-03-29T11:32:00.873 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.866+0000 7f01b02c0640 10 monclient(hunting): picked mon.noname-c con 0x7f01a813ee90 addr v1:172.21.15.135:6790/0 2024-03-29T11:32:00.873 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.866+0000 7f01b02c0640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a8141060 con 0x7f01a80af900 2024-03-29T11:32:00.873 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.866+0000 7f01b02c0640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a8142230 con 0x7f01a80a9ac0 2024-03-29T11:32:00.873 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.866+0000 7f01b02c0640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a81dee80 con 0x7f01a813ee90 2024-03-29T11:32:00.873 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.866+0000 7f01b02c0640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:00.873 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.866+0000 7f01b02c0640 10 monclient(hunting): authenticate will time out at 1228.235748s 2024-03-29T11:32:00.873 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.866+0000 7f01ae85d640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f01a813ee90 0x7f01a813a2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:34702/0 (socket says 172.21.15.135:34702) 2024-03-29T11:32:00.873 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.866+0000 7f01ae85d640 1 -- 172.21.15.135:0/2622676594 learned_addr learned my addr 172.21.15.135:0/2622676594 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:32:00.873 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.866+0000 7f019effd640 10 client.?.objecter ms_handle_connect 0x7f01a813ee90 2024-03-29T11:32:00.874 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.866+0000 7f019effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:00.874 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.867+0000 7f019effd640 10 client.?.objecter ms_handle_connect 0x7f01a80af900 2024-03-29T11:32:00.874 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.867+0000 7f019effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:00.874 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.867+0000 7f019effd640 1 -- 172.21.15.135:0/2622676594 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858777267 0 0) 0x7f01a81dee80 con 0x7f01a813ee90 2024-03-29T11:32:00.874 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.867+0000 7f019effd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:00.874 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.867+0000 7f019effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:00.874 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.867+0000 7f019effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:00.874 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.867+0000 7f019effd640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:32:00.874 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.867+0000 7f019effd640 1 -- 172.21.15.135:0/2622676594 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f018c004eb0 con 0x7f01a813ee90 2024-03-29T11:32:00.874 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.867+0000 7f019effd640 10 client.?.objecter ms_handle_connect 0x7f01a80a9ac0 2024-03-29T11:32:00.875 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.867+0000 7f019effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:00.875 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 1 -- 172.21.15.135:0/2622676594 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3929652560 0 0) 0x7f01a8142230 con 0x7f01a80a9ac0 2024-03-29T11:32:00.875 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:00.875 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:00.875 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:00.875 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:32:00.875 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 1 -- 172.21.15.135:0/2622676594 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01a81dee80 con 0x7f01a80a9ac0 2024-03-29T11:32:00.875 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 1 -- 172.21.15.135:0/2622676594 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 645960131 0 0) 0x7f018c004eb0 con 0x7f01a813ee90 2024-03-29T11:32:00.875 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 1 -- 172.21.15.135:0/2622676594 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01a8142230 con 0x7f01a813ee90 2024-03-29T11:32:00.875 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 1 -- 172.21.15.135:0/2622676594 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f0198001b60 con 0x7f01a813ee90 2024-03-29T11:32:00.875 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:00.875 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:00.875 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 10 monclient(hunting): dump: 2024-03-29T11:32:00.876 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:00.876 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:00.876 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:00.876 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:00.876 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:00.876 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:00.876 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:00.876 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:00.876 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:00.876 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:00.877 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:00.877 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 1 -- 172.21.15.135:0/2622676594 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 884270797 0 0) 0x7f01a81dee80 con 0x7f01a80a9ac0 2024-03-29T11:32:00.877 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 1 -- 172.21.15.135:0/2622676594 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f018c004eb0 con 0x7f01a80a9ac0 2024-03-29T11:32:00.877 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 1 -- 172.21.15.135:0/2622676594 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f01a40018f0 con 0x7f01a80a9ac0 2024-03-29T11:32:00.877 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:00.877 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:00.877 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 10 monclient(hunting): dump: 2024-03-29T11:32:00.877 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:00.877 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:00.877 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:00.877 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:00.877 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:00.877 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:00.877 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:00.878 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:00.878 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:00.878 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:00.878 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 1 -- 172.21.15.135:0/2622676594 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2985941869 0 0) 0x7f01a8141060 con 0x7f01a80af900 2024-03-29T11:32:00.878 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:00.878 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:00.878 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:00.878 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:32:00.878 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 1 -- 172.21.15.135:0/2622676594 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01a81dee80 con 0x7f01a80af900 2024-03-29T11:32:00.878 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 1 -- 172.21.15.135:0/2622676594 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2860434298 0 0) 0x7f018c004eb0 con 0x7f01a80a9ac0 2024-03-29T11:32:00.878 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 1 -- 172.21.15.135:0/2622676594 >> v1:172.21.15.135:6790/0 conn(0x7f01a813ee90 legacy=0x7f01a813a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:00.878 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.868+0000 7f019effd640 1 -- 172.21.15.135:0/2622676594 >> v1:172.21.15.135:6789/0 conn(0x7f01a80af900 legacy=0x7f01a81df3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:00.878 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.869+0000 7f019effd640 10 monclient: _finish_hunting 0 2024-03-29T11:32:00.878 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.869+0000 7f019effd640 1 monclient: found mon.b 2024-03-29T11:32:00.879 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.869+0000 7f019effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:00.879 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.869+0000 7f019effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:32:00.879 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.869+0000 7f019effd640 1 -- 172.21.15.135:0/2622676594 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a81dfab0 con 0x7f01a80a9ac0 2024-03-29T11:32:00.879 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.869+0000 7f019effd640 10 monclient: _finish_auth 0 2024-03-29T11:32:00.879 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.869+0000 7f019effd640 10 monclient: _check_auth_tickets 2024-03-29T11:32:00.879 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.869+0000 7f019effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:00.879 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.869+0000 7f01b02c0640 5 monclient: authenticate success, global_id 4501 2024-03-29T11:32:00.879 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.869+0000 7f01b02c0640 10 monclient: _renew_subs 2024-03-29T11:32:00.879 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.869+0000 7f01b02c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:32:00.879 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.869+0000 7f01b02c0640 1 -- 172.21.15.135:0/2622676594 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a81c82f0 con 0x7f01a80a9ac0 2024-03-29T11:32:00.879 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.869+0000 7f01b02c0640 10 client.4501.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:00.879 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.869+0000 7f01b02c0640 10 monclient: _renew_subs 2024-03-29T11:32:00.879 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.869+0000 7f01b02c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:32:00.879 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.869+0000 7f01b02c0640 1 -- 172.21.15.135:0/2622676594 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a813abd0 con 0x7f01a80a9ac0 2024-03-29T11:32:00.879 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.869+0000 7f01b02c0640 1 librados: init done 2024-03-29T11:32:00.880 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.870+0000 7f01b02c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:00.880 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.870+0000 7f01b02c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:00.880 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.870+0000 7f01b02c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:32:00.880 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.870+0000 7f01b02c0640 1 -- 172.21.15.135:0/2622676594 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a81dece0 con 0x7f01a80a9ac0 2024-03-29T11:32:00.880 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.870+0000 7f019effd640 1 -- 172.21.15.135:0/2622676594 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01a40047e0 con 0x7f01a80a9ac0 2024-03-29T11:32:00.880 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.870+0000 7f019effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:00.880 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.870+0000 7f019effd640 1 -- 172.21.15.135:0/2622676594 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f01a4004950 con 0x7f01a80a9ac0 2024-03-29T11:32:00.880 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.871+0000 7f019effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:00.880 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.871+0000 7f019effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:00.880 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.871+0000 7f019effd640 10 monclient: dump: 2024-03-29T11:32:00.880 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:00.880 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:00.880 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:00.881 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:00.881 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:00.881 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:00.881 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:00.881 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:00.881 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:00.881 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:00.881 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.871+0000 7f019ffff640 4 set_mon_vals no callback set 2024-03-29T11:32:00.881 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.872+0000 7f019effd640 1 -- 172.21.15.135:0/2622676594 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f01a401a920 con 0x7f01a80a9ac0 2024-03-29T11:32:00.881 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.872+0000 7f01ad85b640 10 monclient: get_auth_request con 0x7f018c06df60 auth_method 0 2024-03-29T11:32:00.881 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.872+0000 7f019effd640 1 -- 172.21.15.135:0/2622676594 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f01a40847d0 con 0x7f01a80a9ac0 2024-03-29T11:32:00.881 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.872+0000 7f019effd640 10 client.4501.objecter ms_dispatch 0x7f01a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:00.882 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.872+0000 7f019effd640 3 client.4501.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:00.882 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.872+0000 7f019effd640 3 client.4501.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:00.882 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.872+0000 7f019effd640 20 client.4501.objecter dump_active .. 0 homeless 2024-03-29T11:32:00.882 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.873+0000 7f019effd640 10 client.4501.objecter ms_handle_connect 0x7f018c06df60 2024-03-29T11:32:00.882 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.878+0000 7f019effd640 1 -- 172.21.15.135:0/2622676594 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f01a404fee0 con 0x7f01a80a9ac0 2024-03-29T11:32:00.882 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.878+0000 7f019effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:00.882 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:00.878+0000 7f019effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:01.211 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:01.216+0000 7f01b02c0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:01.211 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:01.216+0000 7f01b02c0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:01.211 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:01.216+0000 7f01b02c0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:32:01.211 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:01.216+0000 7f01b02c0640 1 -- 172.21.15.135:0/2622676594 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01a81dfc40 con 0x7f01a80a9ac0 2024-03-29T11:32:01.212 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:01.217+0000 7f019effd640 1 -- 172.21.15.135:0/2622676594 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f01a4051fe0 con 0x7f01a80a9ac0 2024-03-29T11:32:01.212 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:01.217+0000 7f019effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:01.213 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:01.217+0000 7f019effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:01.213 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:32:01.213 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:01.216 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:01.221+0000 7f01b02c0640 10 librados: watch_flush enter 2024-03-29T11:32:01.217 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:01.221+0000 7f01b02c0640 10 librados: watch_flush exit 2024-03-29T11:32:01.218 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:01.221+0000 7f01b02c0640 20 client.4501.objecter shutdown clearing up homeless session... 2024-03-29T11:32:01.218 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:01.221+0000 7f01b02c0640 10 client.4501.objecter successfully canceled tick 2024-03-29T11:32:01.218 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:01.221+0000 7f01b02c0640 1 -- 172.21.15.135:0/2622676594 >> v1:172.21.15.135:6816/3469875285 conn(0x7f018c06df60 legacy=0x7f018c070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:01.218 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:01.221+0000 7f01b02c0640 10 monclient: shutdown 2024-03-29T11:32:01.218 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:01.221+0000 7f01b02c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:01.218 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:01.221+0000 7f01b02c0640 1 -- 172.21.15.135:0/2622676594 >> v1:172.21.15.194:6789/0 conn(0x7f01a80a9ac0 legacy=0x7f01a8140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:01.218 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:01.221+0000 7f01b02c0640 1 -- 172.21.15.135:0/2622676594 shutdown_connections 2024-03-29T11:32:01.218 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:01.221+0000 7f01b02c0640 1 -- 172.21.15.135:0/2622676594 >> 172.21.15.135:0/2622676594 conn(0x7f01a80b35e0 msgr2=0x7f01a80ab760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:01.218 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:01.221+0000 7f01b02c0640 1 -- 172.21.15.135:0/2622676594 shutdown_connections 2024-03-29T11:32:01.218 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:01.221+0000 7f01b02c0640 1 -- 172.21.15.135:0/2622676594 wait complete. 2024-03-29T11:32:01.219 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:01.221+0000 7f01b02c0640 1 librados: shutdown 2024-03-29T11:32:01.237 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:08.045 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:08.194 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.198+0000 7f1584cac640 10 monclient: get_monmap_and_config 2024-03-29T11:32:08.194 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.198+0000 7f1584cac640 10 monclient: build_initial_monmap 2024-03-29T11:32:08.194 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.198+0000 7f1584cac640 10 monclient: monmap: 2024-03-29T11:32:08.194 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:32:08.194 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:08.194 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:32:08.200455+0000 2024-03-29T11:32:08.194 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:32:08.200455+0000 2024-03-29T11:32:08.195 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:08.195 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:08.195 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:32:08.195 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:32:08.195 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:32:08.195 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:08.195 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.200+0000 7f1584cac640 1 Processor -- start 2024-03-29T11:32:08.195 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.200+0000 7f1584cac640 1 -- start start 2024-03-29T11:32:08.196 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.200+0000 7f1584cac640 10 monclient: init 2024-03-29T11:32:08.196 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.201+0000 7f1584cac640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:08.196 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.201+0000 7f1584cac640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:32:08.196 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.201+0000 7f1584cac640 10 monclient(hunting): picked mon.noname-c con 0x7f15780389e0 addr v1:172.21.15.194:6789/0 2024-03-29T11:32:08.196 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.201+0000 7f1584cac640 10 monclient(hunting): picked mon.noname-a con 0x7f1578038dc0 addr v1:172.21.15.135:6789/0 2024-03-29T11:32:08.196 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.201+0000 7f1584cac640 10 monclient(hunting): picked mon.noname-b con 0x7f1578030660 addr v1:172.21.15.135:6790/0 2024-03-29T11:32:08.197 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.201+0000 7f1584cac640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15780c54c0 con 0x7f1578038dc0 2024-03-29T11:32:08.197 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.201+0000 7f1584cac640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1578036430 con 0x7f15780389e0 2024-03-29T11:32:08.197 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.201+0000 7f1584cac640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1578039840 con 0x7f1578030660 2024-03-29T11:32:08.197 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.201+0000 7f1584cac640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:08.197 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.201+0000 7f1584cac640 10 monclient(hunting): authenticate will time out at 1235.570303s 2024-03-29T11:32:08.197 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.201+0000 7f157effd640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f1578038dc0 0x7f15780cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:55616/0 (socket says 172.21.15.135:55616) 2024-03-29T11:32:08.197 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.201+0000 7f157effd640 1 -- 172.21.15.135:0/2630656900 learned_addr learned my addr 172.21.15.135:0/2630656900 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:32:08.199 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.202+0000 7f157e7fc640 1 -- 172.21.15.135:0/2630656900 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 138055926 0 0) 0x7f15780c54c0 con 0x7f1578038dc0 2024-03-29T11:32:08.199 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.202+0000 7f157e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:32:08.199 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.202+0000 7f157e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:08.199 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.202+0000 7f157e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:08.199 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.202+0000 7f157e7fc640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:32:08.199 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.202+0000 7f157e7fc640 1 -- 172.21.15.135:0/2630656900 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1570003520 con 0x7f1578038dc0 2024-03-29T11:32:08.199 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.202+0000 7f157e7fc640 1 -- 172.21.15.135:0/2630656900 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247096329 0 0) 0x7f1578036430 con 0x7f15780389e0 2024-03-29T11:32:08.199 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.202+0000 7f157e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:32:08.199 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.202+0000 7f157e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:08.199 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.202+0000 7f157e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:08.199 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.202+0000 7f157e7fc640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:32:08.199 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.202+0000 7f157e7fc640 1 -- 172.21.15.135:0/2630656900 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1570003840 con 0x7f15780389e0 2024-03-29T11:32:08.199 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.202+0000 7f157e7fc640 1 -- 172.21.15.135:0/2630656900 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3193235041 0 0) 0x7f1570003520 con 0x7f1578038dc0 2024-03-29T11:32:08.199 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.202+0000 7f157e7fc640 1 -- 172.21.15.135:0/2630656900 >> v1:172.21.15.135:6790/0 conn(0x7f1578030660 legacy=0x7f15780d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:08.200 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.203+0000 7f157e7fc640 1 -- 172.21.15.135:0/2630656900 >> v1:172.21.15.194:6789/0 conn(0x7f15780389e0 legacy=0x7f1578033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:08.200 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.203+0000 7f157e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:32:08.200 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.203+0000 7f157e7fc640 1 monclient: found mon.noname-a 2024-03-29T11:32:08.200 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.203+0000 7f157e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:08.200 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.203+0000 7f157e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.135:6789/0 2024-03-29T11:32:08.200 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.203+0000 7f157e7fc640 1 -- 172.21.15.135:0/2630656900 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1578035690 con 0x7f1578038dc0 2024-03-29T11:32:08.200 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.203+0000 7f157e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:32:08.200 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.203+0000 7f157e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:32:08.200 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.203+0000 7f157e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:08.200 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.203+0000 7f1584cac640 5 monclient: authenticate success, global_id 4491 2024-03-29T11:32:08.200 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.203+0000 7f1584cac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:08.200 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.203+0000 7f157e7fc640 1 -- 172.21.15.135:0/2630656900 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f156c001670 con 0x7f1578038dc0 2024-03-29T11:32:08.200 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.203+0000 7f157e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:08.200 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.203+0000 7f157e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:32:08.200 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.203+0000 7f157e7fc640 10 monclient: dump: 2024-03-29T11:32:08.201 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:08.201 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:08.201 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:08.201 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:08.201 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:08.201 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:08.201 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:08.201 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:08.201 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:08.201 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:08.201 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.203+0000 7f1584cac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:08.201 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.203+0000 7f157e7fc640 1 -- 172.21.15.135:0/2630656900 <== mon.0 v1:172.21.15.135:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f156c001900 con 0x7f1578038dc0 2024-03-29T11:32:08.201 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.203+0000 7f157e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:08.201 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.203+0000 7f1584cac640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:08.202 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.203+0000 7f1584cac640 4 set_mon_vals no callback set 2024-03-29T11:32:08.209 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.204+0000 7f1584cac640 10 monclient: shutdown 2024-03-29T11:32:08.209 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.204+0000 7f1584cac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:08.209 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.204+0000 7f1584cac640 1 -- 172.21.15.135:0/2630656900 >> v1:172.21.15.135:6789/0 conn(0x7f1578038dc0 legacy=0x7f15780cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:08.209 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.204+0000 7f1584cac640 1 -- 172.21.15.135:0/2630656900 shutdown_connections 2024-03-29T11:32:08.209 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.204+0000 7f1584cac640 1 -- 172.21.15.135:0/2630656900 >> 172.21.15.135:0/2630656900 conn(0x7f1578005c00 msgr2=0x7f1578006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:08.210 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.204+0000 7f1584cac640 1 -- 172.21.15.135:0/2630656900 shutdown_connections 2024-03-29T11:32:08.210 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.205+0000 7f1584cac640 1 -- 172.21.15.135:0/2630656900 wait complete. 2024-03-29T11:32:08.210 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.205+0000 7f1584cac640 10 monclient: build_initial_monmap 2024-03-29T11:32:08.210 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.205+0000 7f1584cac640 10 monclient: monmap: 2024-03-29T11:32:08.210 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:32:08.210 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:08.210 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:32:08.210 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:32:08.210 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:08.210 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:08.210 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:32:08.210 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:32:08.210 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:32:08.211 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:08.211 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.206+0000 7f1584cac640 1 librados: starting msgr at 2024-03-29T11:32:08.211 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.206+0000 7f1584cac640 1 librados: starting objecter 2024-03-29T11:32:08.211 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.206+0000 7f1584cac640 1 Processor -- start 2024-03-29T11:32:08.211 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.207+0000 7f1584cac640 1 -- start start 2024-03-29T11:32:08.211 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.207+0000 7f1584cac640 1 librados: setting wanted keys 2024-03-29T11:32:08.211 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.207+0000 7f1584cac640 1 librados: calling monclient init 2024-03-29T11:32:08.211 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.207+0000 7f1584cac640 10 monclient: init 2024-03-29T11:32:08.211 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.207+0000 7f1584cac640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:08.211 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.207+0000 7f1584cac640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:32:08.211 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.207+0000 7f1584cac640 10 monclient(hunting): picked mon.noname-b con 0x7f1578030660 addr v1:172.21.15.194:6789/0 2024-03-29T11:32:08.211 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.207+0000 7f1584cac640 10 monclient(hunting): picked mon.noname-a con 0x7f15780389e0 addr v1:172.21.15.135:6789/0 2024-03-29T11:32:08.211 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.207+0000 7f1584cac640 10 monclient(hunting): picked mon.noname-c con 0x7f15780c5650 addr v1:172.21.15.135:6790/0 2024-03-29T11:32:08.211 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.207+0000 7f1584cac640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1578164750 con 0x7f15780389e0 2024-03-29T11:32:08.212 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.207+0000 7f1584cac640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15780c8640 con 0x7f1578030660 2024-03-29T11:32:08.212 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.207+0000 7f1584cac640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15780c87d0 con 0x7f15780c5650 2024-03-29T11:32:08.212 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.207+0000 7f1584cac640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:08.212 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.207+0000 7f1584cac640 10 monclient(hunting): authenticate will time out at 1235.577001s 2024-03-29T11:32:08.212 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.208+0000 7f157effd640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f15780389e0 0x7f1578164c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:56244/0 (socket says 172.21.15.135:56244) 2024-03-29T11:32:08.212 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.208+0000 7f157ffff640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f15780c5650 0x7f15780c5c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:53958/0 (socket says 172.21.15.135:53958) 2024-03-29T11:32:08.212 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.208+0000 7f157effd640 1 -- 172.21.15.135:0/1998475759 learned_addr learned my addr 172.21.15.135:0/1998475759 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:32:08.212 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.208+0000 7f156bfff640 10 client.?.objecter ms_handle_connect 0x7f15780389e0 2024-03-29T11:32:08.212 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.208+0000 7f156bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:08.212 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.208+0000 7f156bfff640 10 client.?.objecter ms_handle_connect 0x7f15780c5650 2024-03-29T11:32:08.212 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.208+0000 7f156bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:08.213 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.208+0000 7f156bfff640 10 client.?.objecter ms_handle_connect 0x7f1578030660 2024-03-29T11:32:08.213 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.208+0000 7f156bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:08.213 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.208+0000 7f156bfff640 1 -- 172.21.15.135:0/1998475759 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184156730 0 0) 0x7f15780c87d0 con 0x7f15780c5650 2024-03-29T11:32:08.213 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:08.213 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:08.213 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:08.213 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:32:08.213 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 1 -- 172.21.15.135:0/1998475759 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1564004eb0 con 0x7f15780c5650 2024-03-29T11:32:08.213 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 1 -- 172.21.15.135:0/1998475759 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281939580 0 0) 0x7f1578164750 con 0x7f15780389e0 2024-03-29T11:32:08.213 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:08.213 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:08.213 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:08.213 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:32:08.213 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 1 -- 172.21.15.135:0/1998475759 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15780c87d0 con 0x7f15780389e0 2024-03-29T11:32:08.214 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 1 -- 172.21.15.135:0/1998475759 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1559574106 0 0) 0x7f15780c8640 con 0x7f1578030660 2024-03-29T11:32:08.214 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:08.214 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:08.214 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:08.214 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:32:08.214 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 1 -- 172.21.15.135:0/1998475759 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1578164750 con 0x7f1578030660 2024-03-29T11:32:08.214 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 1 -- 172.21.15.135:0/1998475759 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2800873855 0 0) 0x7f1578164750 con 0x7f1578030660 2024-03-29T11:32:08.214 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 1 -- 172.21.15.135:0/1998475759 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15780c8640 con 0x7f1578030660 2024-03-29T11:32:08.214 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 1 -- 172.21.15.135:0/1998475759 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f158004fdd0 con 0x7f1578030660 2024-03-29T11:32:08.214 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:08.214 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:08.215 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 10 monclient(hunting): dump: 2024-03-29T11:32:08.215 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:08.215 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:08.215 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:08.215 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:08.215 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:08.215 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:08.215 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:08.215 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:08.215 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:08.215 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:08.215 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:08.215 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 1 -- 172.21.15.135:0/1998475759 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2411585859 0 0) 0x7f15780c87d0 con 0x7f15780389e0 2024-03-29T11:32:08.215 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 1 -- 172.21.15.135:0/1998475759 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1578164750 con 0x7f15780389e0 2024-03-29T11:32:08.216 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 1 -- 172.21.15.135:0/1998475759 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f156c004980 con 0x7f15780389e0 2024-03-29T11:32:08.216 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:08.216 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:08.216 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.209+0000 7f156bfff640 10 monclient(hunting): dump: 2024-03-29T11:32:08.216 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:08.216 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:08.216 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:08.216 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:08.216 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:08.216 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:08.216 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:08.216 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:08.216 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:08.216 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:08.216 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f156bfff640 1 -- 172.21.15.135:0/1998475759 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3809208385 0 0) 0x7f15780c8640 con 0x7f1578030660 2024-03-29T11:32:08.217 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f156bfff640 1 -- 172.21.15.135:0/1998475759 >> v1:172.21.15.135:6790/0 conn(0x7f15780c5650 legacy=0x7f15780c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:08.217 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f156bfff640 1 -- 172.21.15.135:0/1998475759 >> v1:172.21.15.135:6789/0 conn(0x7f15780389e0 legacy=0x7f1578164c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:08.217 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f156bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:32:08.217 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f156bfff640 1 monclient: found mon.b 2024-03-29T11:32:08.217 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f156bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:08.217 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f156bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:32:08.217 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f156bfff640 1 -- 172.21.15.135:0/1998475759 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1578165380 con 0x7f1578030660 2024-03-29T11:32:08.217 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f156bfff640 10 monclient: _finish_auth 0 2024-03-29T11:32:08.217 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f156bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:32:08.217 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f156bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:08.217 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f156bfff640 1 -- 172.21.15.135:0/1998475759 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f158004f400 con 0x7f1578030660 2024-03-29T11:32:08.217 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f1584cac640 5 monclient: authenticate success, global_id 4507 2024-03-29T11:32:08.217 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f1584cac640 10 monclient: _renew_subs 2024-03-29T11:32:08.217 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f1584cac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:32:08.218 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f1584cac640 1 -- 172.21.15.135:0/1998475759 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15780c0b20 con 0x7f1578030660 2024-03-29T11:32:08.218 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f156bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:08.218 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f156bfff640 1 -- 172.21.15.135:0/1998475759 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f1580075f10 con 0x7f1578030660 2024-03-29T11:32:08.218 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f156bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:08.218 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f156bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:08.218 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f1584cac640 10 client.4507.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:08.218 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f156bfff640 10 monclient: dump: 2024-03-29T11:32:08.218 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:08.218 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:08.218 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:08.218 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:08.218 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:08.218 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:08.218 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:08.219 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:08.219 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:08.219 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:08.219 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f1584cac640 10 monclient: _renew_subs 2024-03-29T11:32:08.219 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f1584cac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:32:08.219 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f1584cac640 1 -- 172.21.15.135:0/1998475759 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15780c6370 con 0x7f1578030660 2024-03-29T11:32:08.219 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f1584cac640 1 librados: init done 2024-03-29T11:32:08.219 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.210+0000 7f157dffb640 4 set_mon_vals no callback set 2024-03-29T11:32:08.219 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.212+0000 7f1584cac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:08.219 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.212+0000 7f1584cac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:08.219 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.212+0000 7f1584cac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:32:08.219 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.212+0000 7f1584cac640 1 -- 172.21.15.135:0/1998475759 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1560005080 con 0x7f1578030660 2024-03-29T11:32:08.219 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.212+0000 7f156bfff640 1 -- 172.21.15.135:0/1998475759 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f1580067bf0 con 0x7f1578030660 2024-03-29T11:32:08.219 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.213+0000 7f157effd640 10 monclient: get_auth_request con 0x7f156406dcb0 auth_method 0 2024-03-29T11:32:08.219 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.213+0000 7f156bfff640 1 -- 172.21.15.135:0/1998475759 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f15800fe270 con 0x7f1578030660 2024-03-29T11:32:08.220 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.213+0000 7f156bfff640 10 client.4507.objecter ms_dispatch 0x7f1580000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:08.220 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.213+0000 7f156bfff640 3 client.4507.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:08.220 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.213+0000 7f156bfff640 3 client.4507.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:08.220 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.213+0000 7f156bfff640 20 client.4507.objecter dump_active .. 0 homeless 2024-03-29T11:32:08.220 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.213+0000 7f156bfff640 10 client.4507.objecter ms_handle_connect 0x7f156406dcb0 2024-03-29T11:32:08.220 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.220+0000 7f156bfff640 1 -- 172.21.15.135:0/1998475759 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f15800c9b50 con 0x7f1578030660 2024-03-29T11:32:08.220 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.220+0000 7f156bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:08.220 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.220+0000 7f156bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:08.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.614+0000 7f1584cac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:08.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.614+0000 7f1584cac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:08.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.614+0000 7f1584cac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:32:08.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.614+0000 7f1584cac640 1 -- 172.21.15.135:0/1998475759 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1560005330 con 0x7f1578030660 2024-03-29T11:32:08.611 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.616+0000 7f156bfff640 1 -- 172.21.15.135:0/1998475759 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f15800cba80 con 0x7f1578030660 2024-03-29T11:32:08.612 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.616+0000 7f156bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:08.612 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.616+0000 7f156bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:08.612 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:32:08.612 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:08.616 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.620+0000 7f1584cac640 10 librados: watch_flush enter 2024-03-29T11:32:08.616 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.620+0000 7f1584cac640 10 librados: watch_flush exit 2024-03-29T11:32:08.616 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.620+0000 7f1584cac640 20 client.4507.objecter shutdown clearing up homeless session... 2024-03-29T11:32:08.616 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.620+0000 7f1584cac640 10 client.4507.objecter successfully canceled tick 2024-03-29T11:32:08.616 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.620+0000 7f1584cac640 1 -- 172.21.15.135:0/1998475759 >> v1:172.21.15.135:6816/3469875285 conn(0x7f156406dcb0 legacy=0x7f1564070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:08.616 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.620+0000 7f1584cac640 10 monclient: shutdown 2024-03-29T11:32:08.616 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.620+0000 7f1584cac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:08.616 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.620+0000 7f1584cac640 1 -- 172.21.15.135:0/1998475759 >> v1:172.21.15.194:6789/0 conn(0x7f1578030660 legacy=0x7f15780c7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:08.616 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.620+0000 7f1584cac640 1 -- 172.21.15.135:0/1998475759 shutdown_connections 2024-03-29T11:32:08.616 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.620+0000 7f1584cac640 1 -- 172.21.15.135:0/1998475759 >> 172.21.15.135:0/1998475759 conn(0x7f1578005c00 msgr2=0x7f1578034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:08.616 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.620+0000 7f1584cac640 1 -- 172.21.15.135:0/1998475759 shutdown_connections 2024-03-29T11:32:08.617 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.620+0000 7f1584cac640 1 -- 172.21.15.135:0/1998475759 wait complete. 2024-03-29T11:32:08.617 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:08.621+0000 7f1584cac640 1 librados: shutdown 2024-03-29T11:32:08.635 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:15.441 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:15.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.582+0000 7f971687b640 10 monclient: get_monmap_and_config 2024-03-29T11:32:15.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.582+0000 7f971687b640 10 monclient: build_initial_monmap 2024-03-29T11:32:15.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.582+0000 7f971687b640 10 monclient: monmap: 2024-03-29T11:32:15.580 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:32:15.580 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:15.580 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:32:15.584160+0000 2024-03-29T11:32:15.580 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:32:15.584160+0000 2024-03-29T11:32:15.580 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:15.580 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:15.581 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:32:15.581 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:32:15.581 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:32:15.581 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:15.581 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.584+0000 7f971687b640 1 Processor -- start 2024-03-29T11:32:15.581 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.584+0000 7f971687b640 1 -- start start 2024-03-29T11:32:15.581 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.584+0000 7f971687b640 10 monclient: init 2024-03-29T11:32:15.581 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.584+0000 7f971687b640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:15.581 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.584+0000 7f971687b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:32:15.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.584+0000 7f971687b640 10 monclient(hunting): picked mon.noname-c con 0x7f97100a9930 addr v1:172.21.15.194:6789/0 2024-03-29T11:32:15.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.584+0000 7f971687b640 10 monclient(hunting): picked mon.noname-a con 0x7f9710149dc0 addr v1:172.21.15.135:6789/0 2024-03-29T11:32:15.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.585+0000 7f970ffff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f97100a9930 0x7f9710143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:48766/0 (socket says 172.21.15.135:48766) 2024-03-29T11:32:15.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.586+0000 7f970ffff640 1 -- 172.21.15.135:0/1022218304 learned_addr learned my addr 172.21.15.135:0/1022218304 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:32:15.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.586+0000 7f971687b640 10 monclient(hunting): picked mon.noname-b con 0x7f971014ff40 addr v1:172.21.15.135:6790/0 2024-03-29T11:32:15.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.586+0000 7f971687b640 1 -- 172.21.15.135:0/1022218304 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97100aae60 con 0x7f9710149dc0 2024-03-29T11:32:15.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.586+0000 7f971687b640 1 -- 172.21.15.135:0/1022218304 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97100b4310 con 0x7f97100a9930 2024-03-29T11:32:15.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.586+0000 7f971687b640 1 -- 172.21.15.135:0/1022218304 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97100afdd0 con 0x7f971014ff40 2024-03-29T11:32:15.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.586+0000 7f971687b640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:15.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.586+0000 7f971687b640 10 monclient(hunting): authenticate will time out at 1242.955474s 2024-03-29T11:32:15.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.586+0000 7f970effd640 1 -- 172.21.15.135:0/1022218304 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4122112451 0 0) 0x7f97100afdd0 con 0x7f971014ff40 2024-03-29T11:32:15.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.586+0000 7f970effd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:15.584 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.586+0000 7f970effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:15.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.586+0000 7f970effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:15.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.586+0000 7f970effd640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:32:15.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.586+0000 7f970effd640 1 -- 172.21.15.135:0/1022218304 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96f8003520 con 0x7f971014ff40 2024-03-29T11:32:15.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.586+0000 7f970effd640 1 -- 172.21.15.135:0/1022218304 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922110587 0 0) 0x7f97100b4310 con 0x7f97100a9930 2024-03-29T11:32:15.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.586+0000 7f970effd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:15.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.586+0000 7f970effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:15.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.586+0000 7f970effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:15.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.586+0000 7f970effd640 10 monclient(hunting): my global_id is 4510 2024-03-29T11:32:15.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.586+0000 7f970effd640 1 -- 172.21.15.135:0/1022218304 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97100afdd0 con 0x7f97100a9930 2024-03-29T11:32:15.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.586+0000 7f970effd640 1 -- 172.21.15.135:0/1022218304 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 938853787 0 0) 0x7f97100afdd0 con 0x7f97100a9930 2024-03-29T11:32:15.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.586+0000 7f970effd640 1 -- 172.21.15.135:0/1022218304 >> v1:172.21.15.135:6790/0 conn(0x7f971014ff40 legacy=0x7f9710152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:15.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.586+0000 7f970effd640 1 -- 172.21.15.135:0/1022218304 >> v1:172.21.15.135:6789/0 conn(0x7f9710149dc0 legacy=0x7f971014a240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:32:15.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.587+0000 7f970effd640 10 monclient: _finish_hunting 0 2024-03-29T11:32:15.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.587+0000 7f970effd640 1 monclient: found mon.noname-c 2024-03-29T11:32:15.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.587+0000 7f970effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:15.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.587+0000 7f970effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:32:15.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.587+0000 7f970effd640 1 -- 172.21.15.135:0/1022218304 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f971014a950 con 0x7f97100a9930 2024-03-29T11:32:15.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.587+0000 7f970effd640 10 monclient: _finish_auth 0 2024-03-29T11:32:15.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.587+0000 7f970effd640 10 monclient: _check_auth_tickets 2024-03-29T11:32:15.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.587+0000 7f970effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:15.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.587+0000 7f970effd640 1 -- 172.21.15.135:0/1022218304 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f97000015a0 con 0x7f97100a9930 2024-03-29T11:32:15.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.587+0000 7f970effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:15.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.587+0000 7f970effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:15.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.587+0000 7f970effd640 10 monclient: dump: 2024-03-29T11:32:15.586 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:15.586 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:15.586 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:15.586 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:15.586 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:15.587 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:15.587 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:15.587 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:15.587 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:15.587 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:15.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.587+0000 7f971687b640 5 monclient: authenticate success, global_id 4510 2024-03-29T11:32:15.592 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.587+0000 7f971687b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:15.592 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.588+0000 7f970effd640 1 -- 172.21.15.135:0/1022218304 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9700001700 con 0x7f97100a9930 2024-03-29T11:32:15.592 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.588+0000 7f970effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:15.592 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.588+0000 7f970effd640 1 -- 172.21.15.135:0/1022218304 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f9700001b50 con 0x7f97100a9930 2024-03-29T11:32:15.592 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.588+0000 7f970effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:15.592 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.588+0000 7f970effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:15.592 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.588+0000 7f970effd640 10 monclient: dump: 2024-03-29T11:32:15.592 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:15.592 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:15.592 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:15.592 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:15.592 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:15.592 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:15.592 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:15.593 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:15.593 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:15.593 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:15.593 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.588+0000 7f971687b640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:15.593 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.588+0000 7f971687b640 4 set_mon_vals no callback set 2024-03-29T11:32:15.593 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.589+0000 7f971687b640 10 monclient: shutdown 2024-03-29T11:32:15.593 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.589+0000 7f971687b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:15.593 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.589+0000 7f971687b640 1 -- 172.21.15.135:0/1022218304 >> v1:172.21.15.194:6789/0 conn(0x7f97100a9930 legacy=0x7f9710143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:15.593 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.589+0000 7f971687b640 1 -- 172.21.15.135:0/1022218304 shutdown_connections 2024-03-29T11:32:15.593 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.589+0000 7f971687b640 1 -- 172.21.15.135:0/1022218304 >> 172.21.15.135:0/1022218304 conn(0x7f97100b35e0 msgr2=0x7f97100b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:15.593 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.589+0000 7f971687b640 1 -- 172.21.15.135:0/1022218304 shutdown_connections 2024-03-29T11:32:15.593 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.589+0000 7f971687b640 1 -- 172.21.15.135:0/1022218304 wait complete. 2024-03-29T11:32:15.593 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.590+0000 7f971687b640 10 monclient: build_initial_monmap 2024-03-29T11:32:15.593 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.590+0000 7f971687b640 10 monclient: monmap: 2024-03-29T11:32:15.593 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:32:15.594 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:15.594 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:32:15.594 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:32:15.594 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:15.594 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:15.594 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:32:15.594 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:32:15.594 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:32:15.594 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:15.594 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.591+0000 7f971687b640 1 librados: starting msgr at 2024-03-29T11:32:15.594 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.591+0000 7f971687b640 1 librados: starting objecter 2024-03-29T11:32:15.594 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.591+0000 7f971687b640 1 Processor -- start 2024-03-29T11:32:15.594 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.591+0000 7f971687b640 1 -- start start 2024-03-29T11:32:15.594 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.591+0000 7f971687b640 1 librados: setting wanted keys 2024-03-29T11:32:15.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.591+0000 7f971687b640 1 librados: calling monclient init 2024-03-29T11:32:15.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.591+0000 7f971687b640 10 monclient: init 2024-03-29T11:32:15.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.592+0000 7f971687b640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:15.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.593+0000 7f971687b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:32:15.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.593+0000 7f971687b640 10 monclient(hunting): picked mon.noname-b con 0x7f97100a9930 addr v1:172.21.15.194:6789/0 2024-03-29T11:32:15.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.593+0000 7f971687b640 10 monclient(hunting): picked mon.noname-a con 0x7f9710149dc0 addr v1:172.21.15.135:6789/0 2024-03-29T11:32:15.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.593+0000 7f971687b640 10 monclient(hunting): picked mon.noname-c con 0x7f971014ff40 addr v1:172.21.15.135:6790/0 2024-03-29T11:32:15.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.593+0000 7f9714e18640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f971014ff40 0x7f9710148120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:58618/0 (socket says 172.21.15.135:58618) 2024-03-29T11:32:15.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.593+0000 7f9714e18640 1 -- 172.21.15.135:0/3609414190 learned_addr learned my addr 172.21.15.135:0/3609414190 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:32:15.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.593+0000 7f971687b640 1 -- 172.21.15.135:0/3609414190 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9710148fc0 con 0x7f9710149dc0 2024-03-29T11:32:15.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.593+0000 7f970cff9640 10 client.?.objecter ms_handle_connect 0x7f971014ff40 2024-03-29T11:32:15.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.593+0000 7f970cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:15.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.593+0000 7f971687b640 1 -- 172.21.15.135:0/3609414190 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9710144a30 con 0x7f97100a9930 2024-03-29T11:32:15.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.593+0000 7f971687b640 1 -- 172.21.15.135:0/3609414190 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9710144510 con 0x7f971014ff40 2024-03-29T11:32:15.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.594+0000 7f971687b640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:15.596 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.594+0000 7f971687b640 10 monclient(hunting): authenticate will time out at 1242.963117s 2024-03-29T11:32:15.596 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.594+0000 7f970cff9640 1 -- 172.21.15.135:0/3609414190 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384220229 0 0) 0x7f9710144510 con 0x7f971014ff40 2024-03-29T11:32:15.596 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.594+0000 7f970cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:32:15.596 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.594+0000 7f970cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:15.596 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.594+0000 7f970cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:15.596 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.594+0000 7f970cff9640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:32:15.596 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.594+0000 7f970cff9640 1 -- 172.21.15.135:0/3609414190 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96fc004eb0 con 0x7f971014ff40 2024-03-29T11:32:15.596 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.594+0000 7f970cff9640 1 -- 172.21.15.135:0/3609414190 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1406324964 0 0) 0x7f96fc004eb0 con 0x7f971014ff40 2024-03-29T11:32:15.596 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.594+0000 7f970cff9640 1 -- 172.21.15.135:0/3609414190 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9710144510 con 0x7f971014ff40 2024-03-29T11:32:15.596 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.594+0000 7f970cff9640 1 -- 172.21.15.135:0/3609414190 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f9708001c30 con 0x7f971014ff40 2024-03-29T11:32:15.596 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.594+0000 7f970cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:15.596 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.594+0000 7f970cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:15.596 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.594+0000 7f970cff9640 10 monclient(hunting): dump: 2024-03-29T11:32:15.596 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:15.597 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:15.597 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:15.597 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:15.597 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:15.597 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:15.597 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:15.597 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:15.597 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:15.597 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:15.597 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.594+0000 7f970cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:15.597 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.594+0000 7f970cff9640 1 -- 172.21.15.135:0/3609414190 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2962001080 0 0) 0x7f9710144510 con 0x7f971014ff40 2024-03-29T11:32:15.597 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.595+0000 7f970cff9640 1 -- 172.21.15.135:0/3609414190 >> v1:172.21.15.194:6789/0 conn(0x7f97100a9930 legacy=0x7f971013b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:15.597 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.595+0000 7f970cff9640 1 -- 172.21.15.135:0/3609414190 >> v1:172.21.15.135:6789/0 conn(0x7f9710149dc0 legacy=0x7f9710147a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:32:15.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.598+0000 7f970cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:32:15.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.598+0000 7f970cff9640 1 monclient: found mon.c 2024-03-29T11:32:15.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.598+0000 7f970cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:15.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.598+0000 7f970cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:32:15.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.598+0000 7f970cff9640 1 -- 172.21.15.135:0/3609414190 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97101446a0 con 0x7f971014ff40 2024-03-29T11:32:15.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.598+0000 7f970cff9640 10 monclient: _finish_auth 0 2024-03-29T11:32:15.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.598+0000 7f970cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:32:15.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.598+0000 7f970cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:15.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.598+0000 7f971687b640 5 monclient: authenticate success, global_id 4526 2024-03-29T11:32:15.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.598+0000 7f971687b640 10 monclient: _renew_subs 2024-03-29T11:32:15.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.598+0000 7f971687b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:32:15.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.598+0000 7f971687b640 1 -- 172.21.15.135:0/3609414190 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f971013b9e0 con 0x7f971014ff40 2024-03-29T11:32:15.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.598+0000 7f971687b640 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:15.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.598+0000 7f971687b640 10 monclient: _renew_subs 2024-03-29T11:32:15.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.598+0000 7f971687b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:32:15.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.598+0000 7f971687b640 1 -- 172.21.15.135:0/3609414190 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9710149750 con 0x7f971014ff40 2024-03-29T11:32:15.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.598+0000 7f971687b640 1 librados: init done 2024-03-29T11:32:15.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.600+0000 7f96f67fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:15.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.600+0000 7f96f67fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:15.601 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.600+0000 7f96f67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:32:15.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.600+0000 7f970cff9640 1 -- 172.21.15.135:0/3609414190 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9708001d90 con 0x7f971014ff40 2024-03-29T11:32:15.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.600+0000 7f96f67fc640 1 -- 172.21.15.135:0/3609414190 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96f0005080 con 0x7f971014ff40 2024-03-29T11:32:15.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.600+0000 7f970cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:15.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.600+0000 7f970cff9640 1 -- 172.21.15.135:0/3609414190 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f9708003c30 con 0x7f971014ff40 2024-03-29T11:32:15.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.601+0000 7f970cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:15.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.601+0000 7f970e7fc640 4 set_mon_vals no callback set 2024-03-29T11:32:15.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.601+0000 7f970cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:15.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.601+0000 7f970cff9640 10 monclient: dump: 2024-03-29T11:32:15.602 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:15.602 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:15.602 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:15.602 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:15.602 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:15.602 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:15.603 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:15.603 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:15.603 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:15.603 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:15.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.601+0000 7f970cff9640 1 -- 172.21.15.135:0/3609414190 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f9708003e50 con 0x7f971014ff40 2024-03-29T11:32:15.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.601+0000 7f970ffff640 10 monclient: get_auth_request con 0x7f96fc06dd20 auth_method 0 2024-03-29T11:32:15.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.602+0000 7f970cff9640 1 -- 172.21.15.135:0/3609414190 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f9708084dc0 con 0x7f971014ff40 2024-03-29T11:32:15.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.602+0000 7f970cff9640 10 client.4526.objecter ms_dispatch 0x7f9710000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:15.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.602+0000 7f970cff9640 3 client.4526.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:15.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.602+0000 7f970cff9640 3 client.4526.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:15.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.602+0000 7f970cff9640 20 client.4526.objecter dump_active .. 0 homeless 2024-03-29T11:32:15.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.602+0000 7f970cff9640 10 client.4526.objecter ms_handle_connect 0x7f96fc06dd20 2024-03-29T11:32:15.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.608+0000 7f970cff9640 1 -- 172.21.15.135:0/3609414190 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9708050760 con 0x7f971014ff40 2024-03-29T11:32:15.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.608+0000 7f970cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:15.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.608+0000 7f970cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:15.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.913+0000 7f96f67fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:15.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.913+0000 7f96f67fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:15.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.913+0000 7f96f67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:32:15.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.913+0000 7f96f67fc640 1 -- 172.21.15.135:0/3609414190 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96f0005330 con 0x7f971014ff40 2024-03-29T11:32:15.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.914+0000 7f970cff9640 1 -- 172.21.15.135:0/3609414190 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f9708052690 con 0x7f971014ff40 2024-03-29T11:32:15.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.915+0000 7f970cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:15.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.915+0000 7f970cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:15.911 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:32:15.911 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:15.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.918+0000 7f96f67fc640 10 librados: watch_flush enter 2024-03-29T11:32:15.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.918+0000 7f96f67fc640 10 librados: watch_flush exit 2024-03-29T11:32:15.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.918+0000 7f96f67fc640 20 client.4526.objecter shutdown clearing up homeless session... 2024-03-29T11:32:15.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.918+0000 7f96f67fc640 10 client.4526.objecter successfully canceled tick 2024-03-29T11:32:15.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.918+0000 7f96f67fc640 1 -- 172.21.15.135:0/3609414190 >> v1:172.21.15.135:6816/3469875285 conn(0x7f96fc06dd20 legacy=0x7f96fc0701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:15.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.918+0000 7f96f67fc640 10 monclient: shutdown 2024-03-29T11:32:15.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.918+0000 7f96f67fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:15.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.918+0000 7f96f67fc640 1 -- 172.21.15.135:0/3609414190 >> v1:172.21.15.135:6790/0 conn(0x7f971014ff40 legacy=0x7f9710148120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:15.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.919+0000 7f96f67fc640 1 -- 172.21.15.135:0/3609414190 shutdown_connections 2024-03-29T11:32:15.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.919+0000 7f96f67fc640 1 -- 172.21.15.135:0/3609414190 >> 172.21.15.135:0/3609414190 conn(0x7f97100b35e0 msgr2=0x7f971014a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:15.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.919+0000 7f96f67fc640 1 -- 172.21.15.135:0/3609414190 shutdown_connections 2024-03-29T11:32:15.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.919+0000 7f96f67fc640 1 -- 172.21.15.135:0/3609414190 wait complete. 2024-03-29T11:32:15.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:15.919+0000 7f96f67fc640 1 librados: shutdown 2024-03-29T11:32:15.929 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:22.737 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:22.885 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.889+0000 7f0135d60640 10 monclient: get_monmap_and_config 2024-03-29T11:32:22.886 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.889+0000 7f0135d60640 10 monclient: build_initial_monmap 2024-03-29T11:32:22.886 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.889+0000 7f0135d60640 10 monclient: monmap: 2024-03-29T11:32:22.886 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:32:22.886 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:22.886 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:32:22.889892+0000 2024-03-29T11:32:22.886 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:32:22.889892+0000 2024-03-29T11:32:22.886 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:22.886 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:22.886 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:32:22.886 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:32:22.886 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:32:22.886 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:22.887 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.891+0000 7f0135d60640 1 Processor -- start 2024-03-29T11:32:22.887 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.891+0000 7f0135d60640 1 -- start start 2024-03-29T11:32:22.887 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.891+0000 7f0135d60640 10 monclient: init 2024-03-29T11:32:22.887 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.892+0000 7f0135d60640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:22.887 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.892+0000 7f0135d60640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:32:22.887 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.892+0000 7f0135d60640 10 monclient(hunting): picked mon.noname-b con 0x7f01300a9ac0 addr v1:172.21.15.135:6790/0 2024-03-29T11:32:22.888 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.892+0000 7f0135d60640 10 monclient(hunting): picked mon.noname-a con 0x7f013013ee90 addr v1:172.21.15.135:6789/0 2024-03-29T11:32:22.888 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.892+0000 7f0135d60640 10 monclient(hunting): picked mon.noname-c con 0x7f01300af900 addr v1:172.21.15.194:6789/0 2024-03-29T11:32:22.888 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.892+0000 7f0135d60640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01300afd10 con 0x7f013013ee90 2024-03-29T11:32:22.888 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.892+0000 7f0135d60640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01300af6f0 con 0x7f01300af900 2024-03-29T11:32:22.888 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.892+0000 7f0135d60640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01300aac80 con 0x7f01300a9ac0 2024-03-29T11:32:22.888 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.892+0000 7f0135d60640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:22.888 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.892+0000 7f0135d60640 10 monclient(hunting): authenticate will time out at 1250.260209s 2024-03-29T11:32:22.895 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.893+0000 7f012f7fe640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f01300a9ac0 0x7f01300adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:44908/0 (socket says 172.21.15.135:44908) 2024-03-29T11:32:22.895 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.893+0000 7f012f7fe640 1 -- 172.21.15.135:0/203262616 learned_addr learned my addr 172.21.15.135:0/203262616 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:32:22.895 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.893+0000 7f012e7fc640 1 -- 172.21.15.135:0/203262616 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1746485154 0 0) 0x7f01300aac80 con 0x7f01300a9ac0 2024-03-29T11:32:22.895 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.893+0000 7f012e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:32:22.895 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.893+0000 7f012e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:22.895 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.893+0000 7f012e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:22.895 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.893+0000 7f012e7fc640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:32:22.896 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.893+0000 7f012e7fc640 1 -- 172.21.15.135:0/203262616 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0118003520 con 0x7f01300a9ac0 2024-03-29T11:32:22.896 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.894+0000 7f012e7fc640 1 -- 172.21.15.135:0/203262616 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 55425474 0 0) 0x7f0118003520 con 0x7f01300a9ac0 2024-03-29T11:32:22.896 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.894+0000 7f012e7fc640 1 -- 172.21.15.135:0/203262616 >> v1:172.21.15.194:6789/0 conn(0x7f01300af900 legacy=0x7f0130149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:32:22.896 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.895+0000 7f012e7fc640 1 -- 172.21.15.135:0/203262616 >> v1:172.21.15.135:6789/0 conn(0x7f013013ee90 legacy=0x7f013013b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:32:22.896 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.895+0000 7f012e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:32:22.896 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.895+0000 7f012e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:32:22.896 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.895+0000 7f012e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:22.896 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.895+0000 7f012e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:32:22.897 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.895+0000 7f012e7fc640 1 -- 172.21.15.135:0/203262616 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01300aaf60 con 0x7f01300a9ac0 2024-03-29T11:32:22.897 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.895+0000 7f012e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:32:22.897 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.895+0000 7f012e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:32:22.897 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.895+0000 7f012e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:22.897 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.895+0000 7f012e7fc640 1 -- 172.21.15.135:0/203262616 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f0124001690 con 0x7f01300a9ac0 2024-03-29T11:32:22.897 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.895+0000 7f012e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:22.897 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.895+0000 7f012e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:22.897 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.895+0000 7f012e7fc640 10 monclient: dump: 2024-03-29T11:32:22.897 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:22.897 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:22.897 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:22.897 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:22.897 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:22.898 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:22.898 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:22.898 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:22.898 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:22.898 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:22.898 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.895+0000 7f0135d60640 5 monclient: authenticate success, global_id 4529 2024-03-29T11:32:22.898 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.895+0000 7f0135d60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:22.898 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.896+0000 7f012e7fc640 1 -- 172.21.15.135:0/203262616 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0124001860 con 0x7f01300a9ac0 2024-03-29T11:32:22.898 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.896+0000 7f012e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:22.898 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.896+0000 7f012e7fc640 1 -- 172.21.15.135:0/203262616 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f0124001cb0 con 0x7f01300a9ac0 2024-03-29T11:32:22.899 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.896+0000 7f0135d60640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:22.899 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.896+0000 7f0135d60640 4 set_mon_vals no callback set 2024-03-29T11:32:22.899 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.896+0000 7f012e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:22.899 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.896+0000 7f012e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:22.899 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.896+0000 7f012e7fc640 10 monclient: dump: 2024-03-29T11:32:22.899 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:22.899 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:22.899 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:22.899 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:22.899 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:22.899 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:22.899 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:22.899 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:22.899 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:22.900 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:22.900 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.897+0000 7f0135d60640 10 monclient: shutdown 2024-03-29T11:32:22.900 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.897+0000 7f0135d60640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:22.900 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.897+0000 7f0135d60640 1 -- 172.21.15.135:0/203262616 >> v1:172.21.15.135:6790/0 conn(0x7f01300a9ac0 legacy=0x7f01300adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:22.900 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.899+0000 7f0135d60640 1 -- 172.21.15.135:0/203262616 shutdown_connections 2024-03-29T11:32:22.900 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.899+0000 7f0135d60640 1 -- 172.21.15.135:0/203262616 >> 172.21.15.135:0/203262616 conn(0x7f01300b35e0 msgr2=0x7f01300b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:22.900 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.900+0000 7f0135d60640 1 -- 172.21.15.135:0/203262616 shutdown_connections 2024-03-29T11:32:22.900 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.900+0000 7f0135d60640 1 -- 172.21.15.135:0/203262616 wait complete. 2024-03-29T11:32:22.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.901+0000 7f0135d60640 10 monclient: build_initial_monmap 2024-03-29T11:32:22.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.901+0000 7f0135d60640 10 monclient: monmap: 2024-03-29T11:32:22.903 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:32:22.903 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:22.903 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:32:22.903 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:32:22.904 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:22.904 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:22.904 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:32:22.904 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:32:22.904 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:32:22.904 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:22.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.901+0000 7f0135d60640 1 librados: starting msgr at 2024-03-29T11:32:22.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.901+0000 7f0135d60640 1 librados: starting objecter 2024-03-29T11:32:22.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.901+0000 7f0135d60640 1 Processor -- start 2024-03-29T11:32:22.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.902+0000 7f0135d60640 1 -- start start 2024-03-29T11:32:22.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.902+0000 7f0135d60640 1 librados: setting wanted keys 2024-03-29T11:32:22.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.902+0000 7f0135d60640 1 librados: calling monclient init 2024-03-29T11:32:22.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.902+0000 7f0135d60640 10 monclient: init 2024-03-29T11:32:22.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.902+0000 7f0135d60640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:22.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.902+0000 7f0135d60640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:32:22.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.903+0000 7f0135d60640 10 monclient(hunting): picked mon.noname-c con 0x7f01300a9ac0 addr v1:172.21.15.135:6790/0 2024-03-29T11:32:22.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.903+0000 7f012f7fe640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f01300a9ac0 0x7f0130141dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:48408/0 (socket says 172.21.15.135:48408) 2024-03-29T11:32:22.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.903+0000 7f012f7fe640 1 -- 172.21.15.135:0/1974081609 learned_addr learned my addr 172.21.15.135:0/1974081609 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:32:22.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.903+0000 7f0135d60640 10 monclient(hunting): picked mon.noname-a con 0x7f01300af900 addr v1:172.21.15.135:6789/0 2024-03-29T11:32:22.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.903+0000 7f0135d60640 10 monclient(hunting): picked mon.noname-b con 0x7f013013ee90 addr v1:172.21.15.194:6789/0 2024-03-29T11:32:22.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.903+0000 7f0135d60640 1 -- 172.21.15.135:0/1974081609 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f013013fed0 con 0x7f01300af900 2024-03-29T11:32:22.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.903+0000 7f0135d60640 1 -- 172.21.15.135:0/1974081609 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0130140380 con 0x7f013013ee90 2024-03-29T11:32:22.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.903+0000 7f0135d60640 1 -- 172.21.15.135:0/1974081609 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01301d30b0 con 0x7f01300a9ac0 2024-03-29T11:32:22.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.903+0000 7f0135d60640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:22.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.903+0000 7f0135d60640 10 monclient(hunting): authenticate will time out at 1250.271501s 2024-03-29T11:32:22.906 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.905+0000 7f0117fff640 10 client.?.objecter ms_handle_connect 0x7f01300a9ac0 2024-03-29T11:32:22.906 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.905+0000 7f0117fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:22.906 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.906+0000 7f0117fff640 1 -- 172.21.15.135:0/1974081609 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562480804 0 0) 0x7f01301d30b0 con 0x7f01300a9ac0 2024-03-29T11:32:22.906 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.906+0000 7f0117fff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:22.907 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.906+0000 7f0117fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:22.907 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.906+0000 7f0117fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:22.907 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.906+0000 7f0117fff640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:32:22.907 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.906+0000 7f0117fff640 1 -- 172.21.15.135:0/1974081609 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f011c004ee0 con 0x7f01300a9ac0 2024-03-29T11:32:22.907 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.906+0000 7f0117fff640 10 client.?.objecter ms_handle_connect 0x7f01300af900 2024-03-29T11:32:22.907 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.906+0000 7f0117fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:22.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.906+0000 7f0117fff640 1 -- 172.21.15.135:0/1974081609 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1006230874 0 0) 0x7f011c004ee0 con 0x7f01300a9ac0 2024-03-29T11:32:22.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.906+0000 7f0117fff640 1 -- 172.21.15.135:0/1974081609 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01301d30b0 con 0x7f01300a9ac0 2024-03-29T11:32:22.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.906+0000 7f0117fff640 1 -- 172.21.15.135:0/1974081609 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f0124001c30 con 0x7f01300a9ac0 2024-03-29T11:32:22.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.906+0000 7f0117fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:22.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.906+0000 7f0117fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:22.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.906+0000 7f0117fff640 10 monclient(hunting): dump: 2024-03-29T11:32:22.908 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:22.908 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:22.908 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:22.908 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:22.908 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:22.908 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:22.908 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:22.908 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:22.909 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:22.909 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:22.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.906+0000 7f0117fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:22.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.906+0000 7f0117fff640 1 -- 172.21.15.135:0/1974081609 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2899065915 0 0) 0x7f013013fed0 con 0x7f01300af900 2024-03-29T11:32:22.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.906+0000 7f0117fff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:22.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.906+0000 7f0117fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:22.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.906+0000 7f0117fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:22.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.906+0000 7f0117fff640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:32:22.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.906+0000 7f0117fff640 1 -- 172.21.15.135:0/1974081609 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f011c004ee0 con 0x7f01300af900 2024-03-29T11:32:22.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.906+0000 7f0117fff640 1 -- 172.21.15.135:0/1974081609 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3094080331 0 0) 0x7f01301d30b0 con 0x7f01300a9ac0 2024-03-29T11:32:22.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.906+0000 7f0117fff640 1 -- 172.21.15.135:0/1974081609 >> v1:172.21.15.194:6789/0 conn(0x7f013013ee90 legacy=0x7f01301de8f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:32:22.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.909+0000 7f0117fff640 1 -- 172.21.15.135:0/1974081609 >> v1:172.21.15.135:6789/0 conn(0x7f01300af900 legacy=0x7f01301d3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:22.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.909+0000 7f0117fff640 10 monclient: _finish_hunting 0 2024-03-29T11:32:22.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.909+0000 7f0117fff640 1 monclient: found mon.c 2024-03-29T11:32:22.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.909+0000 7f0117fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:22.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.909+0000 7f0117fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:32:22.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.909+0000 7f0117fff640 1 -- 172.21.15.135:0/1974081609 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01301398f0 con 0x7f01300a9ac0 2024-03-29T11:32:22.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.909+0000 7f0117fff640 10 monclient: _finish_auth 0 2024-03-29T11:32:22.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.909+0000 7f0117fff640 10 monclient: _check_auth_tickets 2024-03-29T11:32:22.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.909+0000 7f0117fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:22.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.909+0000 7f0135d60640 5 monclient: authenticate success, global_id 4532 2024-03-29T11:32:22.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.909+0000 7f0135d60640 10 monclient: _renew_subs 2024-03-29T11:32:22.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.909+0000 7f0135d60640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:32:22.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.909+0000 7f0135d60640 1 -- 172.21.15.135:0/1974081609 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01301424d0 con 0x7f01300a9ac0 2024-03-29T11:32:22.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.909+0000 7f0135d60640 10 client.4532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:22.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.909+0000 7f0135d60640 10 monclient: _renew_subs 2024-03-29T11:32:22.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.909+0000 7f0135d60640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:32:22.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.909+0000 7f0135d60640 1 -- 172.21.15.135:0/1974081609 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0130072660 con 0x7f01300a9ac0 2024-03-29T11:32:22.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.909+0000 7f0135d60640 1 librados: init done 2024-03-29T11:32:22.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.910+0000 7f0135d60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:22.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.910+0000 7f0135d60640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:22.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.910+0000 7f0135d60640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:32:22.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.910+0000 7f0135d60640 1 -- 172.21.15.135:0/1974081609 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0130141770 con 0x7f01300a9ac0 2024-03-29T11:32:22.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.911+0000 7f0117fff640 1 -- 172.21.15.135:0/1974081609 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0124004a90 con 0x7f01300a9ac0 2024-03-29T11:32:22.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.911+0000 7f0117fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:22.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.911+0000 7f0117fff640 1 -- 172.21.15.135:0/1974081609 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f0124004ea0 con 0x7f01300a9ac0 2024-03-29T11:32:22.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.911+0000 7f0117fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:22.912 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.911+0000 7f0117fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:22.912 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.911+0000 7f0117fff640 10 monclient: dump: 2024-03-29T11:32:22.912 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:22.912 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:22.912 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:22.912 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:22.912 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:22.912 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:22.912 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:22.912 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:22.912 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:22.912 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:22.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.911+0000 7f012dffb640 4 set_mon_vals no callback set 2024-03-29T11:32:22.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.913+0000 7f0117fff640 1 -- 172.21.15.135:0/1974081609 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f012401ae70 con 0x7f01300a9ac0 2024-03-29T11:32:22.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.914+0000 7f0117fff640 1 -- 172.21.15.135:0/1974081609 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f0124084b60 con 0x7f01300a9ac0 2024-03-29T11:32:22.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.914+0000 7f0117fff640 10 client.4532.objecter ms_dispatch 0x7f0130000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:22.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.914+0000 7f0117fff640 3 client.4532.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:22.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.914+0000 7f0117fff640 3 client.4532.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:22.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.914+0000 7f0117fff640 20 client.4532.objecter dump_active .. 0 homeless 2024-03-29T11:32:22.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.914+0000 7f012effd640 10 monclient: get_auth_request con 0x7f011c06db20 auth_method 0 2024-03-29T11:32:22.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.915+0000 7f0117fff640 10 client.4532.objecter ms_handle_connect 0x7f011c06db20 2024-03-29T11:32:22.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.920+0000 7f0117fff640 1 -- 172.21.15.135:0/1974081609 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0124050440 con 0x7f01300a9ac0 2024-03-29T11:32:22.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.920+0000 7f0117fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:22.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:22.920+0000 7f0117fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:23.242 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:23.246+0000 7f0135d60640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:23.242 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:23.246+0000 7f0135d60640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:23.242 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:23.247+0000 7f0135d60640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:32:23.242 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:23.247+0000 7f0135d60640 1 -- 172.21.15.135:0/1974081609 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0130075f10 con 0x7f01300a9ac0 2024-03-29T11:32:23.243 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:23.248+0000 7f0117fff640 1 -- 172.21.15.135:0/1974081609 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f0124052370 con 0x7f01300a9ac0 2024-03-29T11:32:23.243 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:23.248+0000 7f0117fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:23.243 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:23.248+0000 7f0117fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:23.244 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:32:23.244 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:23.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:23.251+0000 7f0135d60640 10 librados: watch_flush enter 2024-03-29T11:32:23.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:23.251+0000 7f0135d60640 10 librados: watch_flush exit 2024-03-29T11:32:23.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:23.251+0000 7f0135d60640 20 client.4532.objecter shutdown clearing up homeless session... 2024-03-29T11:32:23.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:23.251+0000 7f0135d60640 10 client.4532.objecter successfully canceled tick 2024-03-29T11:32:23.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:23.252+0000 7f0135d60640 1 -- 172.21.15.135:0/1974081609 >> v1:172.21.15.135:6816/3469875285 conn(0x7f011c06db20 legacy=0x7f011c06ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:23.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:23.252+0000 7f0135d60640 10 monclient: shutdown 2024-03-29T11:32:23.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:23.252+0000 7f0135d60640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:23.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:23.252+0000 7f0135d60640 1 -- 172.21.15.135:0/1974081609 >> v1:172.21.15.135:6790/0 conn(0x7f01300a9ac0 legacy=0x7f0130141dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:23.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:23.252+0000 7f0135d60640 1 -- 172.21.15.135:0/1974081609 shutdown_connections 2024-03-29T11:32:23.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:23.252+0000 7f0135d60640 1 -- 172.21.15.135:0/1974081609 >> 172.21.15.135:0/1974081609 conn(0x7f01300b35e0 msgr2=0x7f01300b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:23.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:23.252+0000 7f0135d60640 1 -- 172.21.15.135:0/1974081609 shutdown_connections 2024-03-29T11:32:23.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:23.252+0000 7f0135d60640 1 -- 172.21.15.135:0/1974081609 wait complete. 2024-03-29T11:32:23.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:23.252+0000 7f0135d60640 1 librados: shutdown 2024-03-29T11:32:23.267 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:30.074 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:30.228 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.231+0000 7fd5bd0fb640 10 monclient: get_monmap_and_config 2024-03-29T11:32:30.228 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.231+0000 7fd5bd0fb640 10 monclient: build_initial_monmap 2024-03-29T11:32:30.228 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.231+0000 7fd5bd0fb640 10 monclient: monmap: 2024-03-29T11:32:30.228 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:32:30.228 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:30.228 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:32:30.232327+0000 2024-03-29T11:32:30.229 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:32:30.232327+0000 2024-03-29T11:32:30.229 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:30.229 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:30.229 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:32:30.229 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:32:30.229 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:32:30.229 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:30.229 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.232+0000 7fd5bd0fb640 1 Processor -- start 2024-03-29T11:32:30.229 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.232+0000 7fd5bd0fb640 1 -- start start 2024-03-29T11:32:30.229 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.232+0000 7fd5bd0fb640 10 monclient: init 2024-03-29T11:32:30.232 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.233+0000 7fd5bd0fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:30.232 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.233+0000 7fd5bd0fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:32:30.232 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.233+0000 7fd5bd0fb640 10 monclient(hunting): picked mon.noname-a con 0x7fd5b80aaee0 addr v1:172.21.15.135:6789/0 2024-03-29T11:32:30.232 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.233+0000 7fd5bd0fb640 10 monclient(hunting): picked mon.noname-c con 0x7fd5b80ae460 addr v1:172.21.15.194:6789/0 2024-03-29T11:32:30.232 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.233+0000 7fd5bd0fb640 10 monclient(hunting): picked mon.noname-b con 0x7fd5b80aeff0 addr v1:172.21.15.135:6790/0 2024-03-29T11:32:30.232 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.233+0000 7fd5bd0fb640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b80abb60 con 0x7fd5b80aaee0 2024-03-29T11:32:30.232 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.233+0000 7fd5bd0fb640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b80abd90 con 0x7fd5b80ae460 2024-03-29T11:32:30.232 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.233+0000 7fd5bd0fb640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b80ac070 con 0x7fd5b80aeff0 2024-03-29T11:32:30.232 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.233+0000 7fd5bd0fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:30.233 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.233+0000 7fd5bd0fb640 10 monclient(hunting): authenticate will time out at 1257.601731s 2024-03-29T11:32:30.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.240+0000 7fd5b77fe640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fd5b80ae460 0x7fd5b80ae8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:33851/0 (socket says 172.21.15.135:33851) 2024-03-29T11:32:30.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.241+0000 7fd5b77fe640 1 -- 172.21.15.135:0/2932446137 learned_addr learned my addr 172.21.15.135:0/2932446137 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:32:30.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.241+0000 7fd5b6ffd640 1 -- 172.21.15.135:0/2932446137 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814431523 0 0) 0x7fd5b80abd90 con 0x7fd5b80ae460 2024-03-29T11:32:30.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.241+0000 7fd5b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:30.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.241+0000 7fd5b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:30.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.241+0000 7fd5b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:30.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.241+0000 7fd5b6ffd640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:32:30.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.241+0000 7fd5b6ffd640 1 -- 172.21.15.135:0/2932446137 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5a0003520 con 0x7fd5b80ae460 2024-03-29T11:32:30.241 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 1 -- 172.21.15.135:0/2932446137 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3545412347 0 0) 0x7fd5b80ac070 con 0x7fd5b80aeff0 2024-03-29T11:32:30.241 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:32:30.241 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:30.241 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:30.241 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:32:30.241 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 1 -- 172.21.15.135:0/2932446137 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5b80abd90 con 0x7fd5b80aeff0 2024-03-29T11:32:30.241 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 1 -- 172.21.15.135:0/2932446137 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4280187579 0 0) 0x7fd5a0003520 con 0x7fd5b80ae460 2024-03-29T11:32:30.241 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 1 -- 172.21.15.135:0/2932446137 >> v1:172.21.15.135:6790/0 conn(0x7fd5b80aeff0 legacy=0x7fd5b80af480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:30.241 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 1 -- 172.21.15.135:0/2932446137 >> v1:172.21.15.135:6789/0 conn(0x7fd5b80aaee0 legacy=0x7fd5b80acfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:30.241 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:32:30.241 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 1 monclient: found mon.noname-c 2024-03-29T11:32:30.242 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:30.242 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:32:30.242 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 1 -- 172.21.15.135:0/2932446137 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b80ac350 con 0x7fd5b80ae460 2024-03-29T11:32:30.242 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:32:30.242 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:32:30.242 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:30.242 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 1 -- 172.21.15.135:0/2932446137 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fd5a8004c90 con 0x7fd5b80ae460 2024-03-29T11:32:30.242 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:30.243 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:30.243 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 10 monclient: dump: 2024-03-29T11:32:30.243 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:30.243 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:30.243 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:30.243 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:30.244 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:30.244 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:30.244 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:30.244 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:30.244 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:30.245 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:30.245 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5bd0fb640 5 monclient: authenticate success, global_id 4513 2024-03-29T11:32:30.245 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5bd0fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:30.245 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 1 -- 172.21.15.135:0/2932446137 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5a8004df0 con 0x7fd5b80ae460 2024-03-29T11:32:30.245 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:30.245 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 1 -- 172.21.15.135:0/2932446137 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fd5a8001410 con 0x7fd5b80ae460 2024-03-29T11:32:30.245 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:30.245 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:30.246 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5b6ffd640 10 monclient: dump: 2024-03-29T11:32:30.246 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:30.246 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:30.246 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:30.246 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:30.246 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:30.246 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:30.246 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:30.246 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:30.246 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:30.247 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:30.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5bd0fb640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:30.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.242+0000 7fd5bd0fb640 4 set_mon_vals no callback set 2024-03-29T11:32:30.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.243+0000 7fd5bd0fb640 10 monclient: shutdown 2024-03-29T11:32:30.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.243+0000 7fd5bd0fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:30.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.243+0000 7fd5bd0fb640 1 -- 172.21.15.135:0/2932446137 >> v1:172.21.15.194:6789/0 conn(0x7fd5b80ae460 legacy=0x7fd5b80ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:30.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.243+0000 7fd5bd0fb640 1 -- 172.21.15.135:0/2932446137 shutdown_connections 2024-03-29T11:32:30.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.243+0000 7fd5bd0fb640 1 -- 172.21.15.135:0/2932446137 >> 172.21.15.135:0/2932446137 conn(0x7fd5b80b3a30 msgr2=0x7fd5b80b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:30.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.243+0000 7fd5bd0fb640 1 -- 172.21.15.135:0/2932446137 shutdown_connections 2024-03-29T11:32:30.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.243+0000 7fd5bd0fb640 1 -- 172.21.15.135:0/2932446137 wait complete. 2024-03-29T11:32:30.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.244+0000 7fd5bd0fb640 10 monclient: build_initial_monmap 2024-03-29T11:32:30.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.244+0000 7fd5bd0fb640 10 monclient: monmap: 2024-03-29T11:32:30.248 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:32:30.248 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:30.248 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:32:30.248 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:32:30.248 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:30.248 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:30.248 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:32:30.248 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:32:30.248 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:32:30.248 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:30.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.245+0000 7fd5bd0fb640 1 librados: starting msgr at 2024-03-29T11:32:30.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.245+0000 7fd5bd0fb640 1 librados: starting objecter 2024-03-29T11:32:30.249 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.245+0000 7fd5bd0fb640 1 Processor -- start 2024-03-29T11:32:30.252 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.245+0000 7fd5bd0fb640 1 -- start start 2024-03-29T11:32:30.252 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.245+0000 7fd5bd0fb640 1 librados: setting wanted keys 2024-03-29T11:32:30.252 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.245+0000 7fd5bd0fb640 1 librados: calling monclient init 2024-03-29T11:32:30.252 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.245+0000 7fd5bd0fb640 10 monclient: init 2024-03-29T11:32:30.252 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.246+0000 7fd5bd0fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:30.253 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.246+0000 7fd5bd0fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:32:30.253 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.247+0000 7fd5bd0fb640 10 monclient(hunting): picked mon.noname-c con 0x7fd5b80aaee0 addr v1:172.21.15.135:6790/0 2024-03-29T11:32:30.253 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.247+0000 7fd5bd0fb640 10 monclient(hunting): picked mon.noname-a con 0x7fd5b80ae460 addr v1:172.21.15.135:6789/0 2024-03-29T11:32:30.253 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.247+0000 7fd5bd0fb640 10 monclient(hunting): picked mon.noname-b con 0x7fd5b80aeff0 addr v1:172.21.15.194:6789/0 2024-03-29T11:32:30.253 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.247+0000 7fd5bd0fb640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b8070040 con 0x7fd5b80ae460 2024-03-29T11:32:30.253 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.247+0000 7fd5bd0fb640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b8070320 con 0x7fd5b80aeff0 2024-03-29T11:32:30.253 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.247+0000 7fd5bd0fb640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b806ea10 con 0x7fd5b80aaee0 2024-03-29T11:32:30.253 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.247+0000 7fd5bd0fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:30.253 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.247+0000 7fd5bd0fb640 10 monclient(hunting): authenticate will time out at 1257.615329s 2024-03-29T11:32:30.253 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.255+0000 7fd5b7fff640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7fd5b80aaee0 0x7fd5b81efcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:57725/0 (socket says 172.21.15.135:57725) 2024-03-29T11:32:30.253 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.255+0000 7fd5b7fff640 1 -- 172.21.15.135:0/437964330 learned_addr learned my addr 172.21.15.135:0/437964330 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:32:30.253 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.255+0000 7fd5b4ff9640 10 client.?.objecter ms_handle_connect 0x7fd5b80aaee0 2024-03-29T11:32:30.253 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.255+0000 7fd5b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:30.253 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.255+0000 7fd5b4ff9640 1 -- 172.21.15.135:0/437964330 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889697769 0 0) 0x7fd5b806ea10 con 0x7fd5b80aaee0 2024-03-29T11:32:30.253 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.255+0000 7fd5b4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:32:30.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.255+0000 7fd5b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:30.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.255+0000 7fd5b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:30.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.255+0000 7fd5b4ff9640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:32:30.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.255+0000 7fd5b4ff9640 1 -- 172.21.15.135:0/437964330 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5a4004eb0 con 0x7fd5b80aaee0 2024-03-29T11:32:30.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.255+0000 7fd5b4ff9640 1 -- 172.21.15.135:0/437964330 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2684260918 0 0) 0x7fd5a4004eb0 con 0x7fd5b80aaee0 2024-03-29T11:32:30.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.255+0000 7fd5b4ff9640 1 -- 172.21.15.135:0/437964330 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5b806ea10 con 0x7fd5b80aaee0 2024-03-29T11:32:30.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.255+0000 7fd5b4ff9640 1 -- 172.21.15.135:0/437964330 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fd5ac001960 con 0x7fd5b80aaee0 2024-03-29T11:32:30.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.255+0000 7fd5b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:30.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.255+0000 7fd5b4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:30.254 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.255+0000 7fd5b4ff9640 10 monclient(hunting): dump: 2024-03-29T11:32:30.254 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:30.254 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:30.254 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:30.255 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:30.255 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:30.255 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:30.255 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:30.255 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:30.255 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:30.255 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:30.255 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.255+0000 7fd5b4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:30.255 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.256+0000 7fd5b4ff9640 1 -- 172.21.15.135:0/437964330 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3089835372 0 0) 0x7fd5b806ea10 con 0x7fd5b80aaee0 2024-03-29T11:32:30.255 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.256+0000 7fd5b4ff9640 1 -- 172.21.15.135:0/437964330 >> v1:172.21.15.194:6789/0 conn(0x7fd5b80aeff0 legacy=0x7fd5b81f6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:30.255 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.256+0000 7fd5b4ff9640 1 -- 172.21.15.135:0/437964330 >> v1:172.21.15.135:6789/0 conn(0x7fd5b80ae460 legacy=0x7fd5b81f03e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:32:30.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.257+0000 7fd5b4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:32:30.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.257+0000 7fd5b4ff9640 1 monclient: found mon.c 2024-03-29T11:32:30.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.257+0000 7fd5b4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:30.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.257+0000 7fd5b4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:32:30.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.257+0000 7fd5b4ff9640 1 -- 172.21.15.135:0/437964330 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b806f1a0 con 0x7fd5b80aaee0 2024-03-29T11:32:30.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.257+0000 7fd5b4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:32:30.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.257+0000 7fd5b4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:32:30.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.257+0000 7fd5b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:30.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.257+0000 7fd5bd0fb640 5 monclient: authenticate success, global_id 4538 2024-03-29T11:32:30.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.257+0000 7fd5bd0fb640 10 monclient: _renew_subs 2024-03-29T11:32:30.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.257+0000 7fd5bd0fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:32:30.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.257+0000 7fd5bd0fb640 1 -- 172.21.15.135:0/437964330 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b806dce0 con 0x7fd5b80aaee0 2024-03-29T11:32:30.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.257+0000 7fd5b4ff9640 1 -- 172.21.15.135:0/437964330 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5ac001c80 con 0x7fd5b80aaee0 2024-03-29T11:32:30.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.257+0000 7fd5b4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:30.260 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.257+0000 7fd5b4ff9640 1 -- 172.21.15.135:0/437964330 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fd5ac003a40 con 0x7fd5b80aaee0 2024-03-29T11:32:30.260 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.257+0000 7fd5b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:30.260 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.257+0000 7fd5b4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:30.260 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.257+0000 7fd5b4ff9640 10 monclient: dump: 2024-03-29T11:32:30.260 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:30.260 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:30.260 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:30.260 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:30.260 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:30.260 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:30.260 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:30.260 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:30.260 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:30.261 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:30.261 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.257+0000 7fd5b67fc640 4 set_mon_vals no callback set 2024-03-29T11:32:30.261 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.261+0000 7fd5bd0fb640 10 client.4538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:30.261 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.261+0000 7fd5bd0fb640 10 monclient: _renew_subs 2024-03-29T11:32:30.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.261+0000 7fd5bd0fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:32:30.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.261+0000 7fd5bd0fb640 1 -- 172.21.15.135:0/437964330 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b814d570 con 0x7fd5b80aaee0 2024-03-29T11:32:30.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.261+0000 7fd5bd0fb640 1 librados: init done 2024-03-29T11:32:30.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.263+0000 7fd5b4ff9640 1 -- 172.21.15.135:0/437964330 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fd5ac003ba0 con 0x7fd5b80aaee0 2024-03-29T11:32:30.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.263+0000 7fd5b4ff9640 1 -- 172.21.15.135:0/437964330 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7fd5ac084b30 con 0x7fd5b80aaee0 2024-03-29T11:32:30.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.263+0000 7fd5b4ff9640 10 client.4538.objecter ms_dispatch 0x7fd5b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:30.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.263+0000 7fd5b4ff9640 3 client.4538.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:30.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.263+0000 7fd5b4ff9640 3 client.4538.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:30.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.264+0000 7fd5b4ff9640 20 client.4538.objecter dump_active .. 0 homeless 2024-03-29T11:32:30.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.265+0000 7fd5b77fe640 10 monclient: get_auth_request con 0x7fd5a406dcd0 auth_method 0 2024-03-29T11:32:30.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.265+0000 7fd5b4ff9640 10 client.4538.objecter ms_handle_connect 0x7fd5a406dcd0 2024-03-29T11:32:30.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.266+0000 7fd5bd0fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:30.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.266+0000 7fd5bd0fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:30.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.266+0000 7fd5bd0fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:32:30.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.266+0000 7fd5bd0fb640 1 -- 172.21.15.135:0/437964330 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5b806f330 con 0x7fd5b80aaee0 2024-03-29T11:32:30.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.275+0000 7fd5b4ff9640 1 -- 172.21.15.135:0/437964330 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd5ac050430 con 0x7fd5b80aaee0 2024-03-29T11:32:30.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.275+0000 7fd5b4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:30.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.275+0000 7fd5b4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:30.596 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.601+0000 7fd5bd0fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:30.596 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.601+0000 7fd5bd0fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:30.596 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.601+0000 7fd5bd0fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:32:30.596 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.601+0000 7fd5bd0fb640 1 -- 172.21.15.135:0/437964330 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5b814d0b0 con 0x7fd5b80aaee0 2024-03-29T11:32:30.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.605+0000 7fd5b4ff9640 1 -- 172.21.15.135:0/437964330 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7fd5b814d0b0 con 0x7fd5b80aaee0 2024-03-29T11:32:30.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.605+0000 7fd5b4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:30.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.605+0000 7fd5b4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:30.601 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:32:30.601 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:30.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.608+0000 7fd5bd0fb640 10 librados: watch_flush enter 2024-03-29T11:32:30.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.608+0000 7fd5bd0fb640 10 librados: watch_flush exit 2024-03-29T11:32:30.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.608+0000 7fd5bd0fb640 20 client.4538.objecter shutdown clearing up homeless session... 2024-03-29T11:32:30.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.608+0000 7fd5bd0fb640 10 client.4538.objecter successfully canceled tick 2024-03-29T11:32:30.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.610+0000 7fd5bd0fb640 1 -- 172.21.15.135:0/437964330 >> v1:172.21.15.135:6816/3469875285 conn(0x7fd5a406dcd0 legacy=0x7fd5a4070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:30.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.610+0000 7fd5bd0fb640 10 monclient: shutdown 2024-03-29T11:32:30.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.610+0000 7fd5bd0fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:30.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.610+0000 7fd5bd0fb640 1 -- 172.21.15.135:0/437964330 >> v1:172.21.15.135:6790/0 conn(0x7fd5b80aaee0 legacy=0x7fd5b81efcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:30.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.612+0000 7fd5bd0fb640 1 -- 172.21.15.135:0/437964330 shutdown_connections 2024-03-29T11:32:30.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.612+0000 7fd5bd0fb640 1 -- 172.21.15.135:0/437964330 >> 172.21.15.135:0/437964330 conn(0x7fd5b80b3a30 msgr2=0x7fd5b81517e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:30.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.612+0000 7fd5bd0fb640 1 -- 172.21.15.135:0/437964330 shutdown_connections 2024-03-29T11:32:30.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.612+0000 7fd5bd0fb640 1 -- 172.21.15.135:0/437964330 wait complete. 2024-03-29T11:32:30.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:30.612+0000 7fd5bd0fb640 1 librados: shutdown 2024-03-29T11:32:30.627 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:37.434 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:37.594 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.598+0000 7f2049863640 10 monclient: get_monmap_and_config 2024-03-29T11:32:37.594 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.598+0000 7f2049863640 10 monclient: build_initial_monmap 2024-03-29T11:32:37.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.598+0000 7f2049863640 10 monclient: monmap: 2024-03-29T11:32:37.595 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:32:37.595 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:37.595 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:32:37.598762+0000 2024-03-29T11:32:37.595 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:32:37.598762+0000 2024-03-29T11:32:37.595 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:37.595 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:37.595 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:32:37.595 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:32:37.595 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:32:37.595 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:37.596 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.601+0000 7f2049863640 1 Processor -- start 2024-03-29T11:32:37.596 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.601+0000 7f2049863640 1 -- start start 2024-03-29T11:32:37.596 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.601+0000 7f2049863640 10 monclient: init 2024-03-29T11:32:37.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.601+0000 7f2049863640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:37.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.601+0000 7f2049863640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:32:37.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.601+0000 7f2049863640 10 monclient(hunting): picked mon.noname-c con 0x7f20440a9930 addr v1:172.21.15.194:6789/0 2024-03-29T11:32:37.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.601+0000 7f2049863640 10 monclient(hunting): picked mon.noname-a con 0x7f2044149dc0 addr v1:172.21.15.135:6789/0 2024-03-29T11:32:37.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.604+0000 7f2049863640 10 monclient(hunting): picked mon.noname-b con 0x7f204414ff40 addr v1:172.21.15.135:6790/0 2024-03-29T11:32:37.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.604+0000 7f2049863640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20440aae60 con 0x7f2044149dc0 2024-03-29T11:32:37.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.604+0000 7f2049863640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20440b4310 con 0x7f20440a9930 2024-03-29T11:32:37.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.604+0000 7f2049863640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20440afdd0 con 0x7f204414ff40 2024-03-29T11:32:37.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.604+0000 7f2049863640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:37.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.604+0000 7f2049863640 10 monclient(hunting): authenticate will time out at 1264.972211s 2024-03-29T11:32:37.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.606+0000 7f20427fc640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f2044149dc0 0x7f204414a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:33887/0 (socket says 172.21.15.135:33887) 2024-03-29T11:32:37.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.606+0000 7f20427fc640 1 -- 172.21.15.135:0/1684767761 learned_addr learned my addr 172.21.15.135:0/1684767761 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:32:37.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.608+0000 7f2041ffb640 1 -- 172.21.15.135:0/1684767761 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580214807 0 0) 0x7f20440aae60 con 0x7f2044149dc0 2024-03-29T11:32:37.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.608+0000 7f2041ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:37.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.608+0000 7f2041ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:37.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.608+0000 7f2041ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:37.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.608+0000 7f2041ffb640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:32:37.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.608+0000 7f2041ffb640 1 -- 172.21.15.135:0/1684767761 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f202c003520 con 0x7f2044149dc0 2024-03-29T11:32:37.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.609+0000 7f2041ffb640 1 -- 172.21.15.135:0/1684767761 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 510152575 0 0) 0x7f202c003520 con 0x7f2044149dc0 2024-03-29T11:32:37.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.609+0000 7f2041ffb640 1 -- 172.21.15.135:0/1684767761 >> v1:172.21.15.135:6790/0 conn(0x7f204414ff40 legacy=0x7f2044152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:37.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.609+0000 7f2041ffb640 1 -- 172.21.15.135:0/1684767761 >> v1:172.21.15.194:6789/0 conn(0x7f20440a9930 legacy=0x7f2044143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:32:37.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.609+0000 7f2041ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:32:37.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.609+0000 7f2041ffb640 1 monclient: found mon.noname-a 2024-03-29T11:32:37.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.609+0000 7f2041ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:37.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.609+0000 7f2041ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.135:6789/0 2024-03-29T11:32:37.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.609+0000 7f2041ffb640 1 -- 172.21.15.135:0/1684767761 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204414a950 con 0x7f2044149dc0 2024-03-29T11:32:37.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.609+0000 7f2041ffb640 10 monclient: _finish_auth 0 2024-03-29T11:32:37.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.609+0000 7f2041ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:32:37.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.609+0000 7f2041ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:37.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.609+0000 7f2041ffb640 1 -- 172.21.15.135:0/1684767761 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f2038001690 con 0x7f2044149dc0 2024-03-29T11:32:37.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.609+0000 7f2041ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:37.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.609+0000 7f2041ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:32:37.606 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.609+0000 7f2041ffb640 10 monclient: dump: 2024-03-29T11:32:37.606 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:37.606 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:37.606 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:37.606 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:37.606 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:37.606 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:37.606 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:37.606 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:37.606 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:37.607 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:37.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.609+0000 7f2041ffb640 1 -- 172.21.15.135:0/1684767761 <== mon.0 v1:172.21.15.135:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2038001920 con 0x7f2044149dc0 2024-03-29T11:32:37.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.609+0000 7f2041ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:37.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.609+0000 7f2041ffb640 1 -- 172.21.15.135:0/1684767761 <== mon.0 v1:172.21.15.135:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f2038001cb0 con 0x7f2044149dc0 2024-03-29T11:32:37.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.610+0000 7f2041ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:37.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.610+0000 7f2041ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:37.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.610+0000 7f2041ffb640 10 monclient: dump: 2024-03-29T11:32:37.607 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:37.607 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:37.607 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:37.607 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:37.607 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:37.607 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:37.608 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:37.608 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:37.608 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:37.608 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:37.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.610+0000 7f2049863640 5 monclient: authenticate success, global_id 4500 2024-03-29T11:32:37.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.611+0000 7f2049863640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:37.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.611+0000 7f2049863640 4 set_mon_vals no callback set 2024-03-29T11:32:37.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.612+0000 7f2049863640 10 monclient: shutdown 2024-03-29T11:32:37.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.612+0000 7f2049863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:37.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.612+0000 7f2049863640 1 -- 172.21.15.135:0/1684767761 >> v1:172.21.15.135:6789/0 conn(0x7f2044149dc0 legacy=0x7f204414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:37.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.612+0000 7f2049863640 1 -- 172.21.15.135:0/1684767761 shutdown_connections 2024-03-29T11:32:37.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.612+0000 7f2049863640 1 -- 172.21.15.135:0/1684767761 >> 172.21.15.135:0/1684767761 conn(0x7f20440b35e0 msgr2=0x7f20440b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:37.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.613+0000 7f2049863640 1 -- 172.21.15.135:0/1684767761 shutdown_connections 2024-03-29T11:32:37.611 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.615+0000 7f2049863640 1 -- 172.21.15.135:0/1684767761 wait complete. 2024-03-29T11:32:37.614 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.616+0000 7f2049863640 10 monclient: build_initial_monmap 2024-03-29T11:32:37.614 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.617+0000 7f2049863640 10 monclient: monmap: 2024-03-29T11:32:37.614 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:32:37.614 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:37.614 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:32:37.614 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:32:37.614 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:37.615 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:37.615 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:32:37.615 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:32:37.615 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:32:37.615 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:37.615 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.617+0000 7f2049863640 1 librados: starting msgr at 2024-03-29T11:32:37.615 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.617+0000 7f2049863640 1 librados: starting objecter 2024-03-29T11:32:37.615 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.617+0000 7f2049863640 1 Processor -- start 2024-03-29T11:32:37.615 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.618+0000 7f2049863640 1 -- start start 2024-03-29T11:32:37.615 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.618+0000 7f2049863640 1 librados: setting wanted keys 2024-03-29T11:32:37.615 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.618+0000 7f2049863640 1 librados: calling monclient init 2024-03-29T11:32:37.615 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.618+0000 7f2049863640 10 monclient: init 2024-03-29T11:32:37.615 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.619+0000 7f2049863640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:37.615 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.619+0000 7f2049863640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:32:37.616 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.619+0000 7f2049863640 10 monclient(hunting): picked mon.noname-a con 0x7f20440a9930 addr v1:172.21.15.135:6789/0 2024-03-29T11:32:37.619 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.624+0000 7f2042ffd640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f20440a9930 0x7f20441382c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:34053/0 (socket says 172.21.15.135:34053) 2024-03-29T11:32:37.619 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.624+0000 7f2049863640 10 monclient(hunting): picked mon.noname-b con 0x7f2044149dc0 addr v1:172.21.15.194:6789/0 2024-03-29T11:32:37.619 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.624+0000 7f2042ffd640 1 -- 172.21.15.135:0/3307403794 learned_addr learned my addr 172.21.15.135:0/3307403794 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:32:37.627 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.628+0000 7f2049863640 10 monclient(hunting): picked mon.noname-c con 0x7f204414ff40 addr v1:172.21.15.135:6790/0 2024-03-29T11:32:37.627 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.628+0000 7f2049863640 1 -- 172.21.15.135:0/3307403794 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f204413ea80 con 0x7f20440a9930 2024-03-29T11:32:37.627 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.628+0000 7f2049863640 1 -- 172.21.15.135:0/3307403794 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f204413ec10 con 0x7f2044149dc0 2024-03-29T11:32:37.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.628+0000 7f202bfff640 10 client.?.objecter ms_handle_connect 0x7f2044149dc0 2024-03-29T11:32:37.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.628+0000 7f202bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:37.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.628+0000 7f2049863640 1 -- 172.21.15.135:0/3307403794 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2044148fb0 con 0x7f204414ff40 2024-03-29T11:32:37.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.628+0000 7f2049863640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:37.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.628+0000 7f2049863640 10 monclient(hunting): authenticate will time out at 1264.996644s 2024-03-29T11:32:37.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.628+0000 7f202bfff640 10 client.?.objecter ms_handle_connect 0x7f20440a9930 2024-03-29T11:32:37.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.628+0000 7f202bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:37.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.628+0000 7f202bfff640 1 -- 172.21.15.135:0/3307403794 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920377683 0 0) 0x7f204413ea80 con 0x7f20440a9930 2024-03-29T11:32:37.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.628+0000 7f202bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:37.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.628+0000 7f202bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:37.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.628+0000 7f202bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:37.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.628+0000 7f202bfff640 10 monclient(hunting): my global_id is 4503 2024-03-29T11:32:37.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.628+0000 7f202bfff640 1 -- 172.21.15.135:0/3307403794 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2030004eb0 con 0x7f20440a9930 2024-03-29T11:32:37.628 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.629+0000 7f202bfff640 1 -- 172.21.15.135:0/3307403794 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3993686620 0 0) 0x7f2030004eb0 con 0x7f20440a9930 2024-03-29T11:32:37.629 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.629+0000 7f202bfff640 1 -- 172.21.15.135:0/3307403794 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2030005300 con 0x7f20440a9930 2024-03-29T11:32:37.629 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.629+0000 7f202bfff640 1 -- 172.21.15.135:0/3307403794 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f2034004040 con 0x7f20440a9930 2024-03-29T11:32:37.629 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.629+0000 7f202bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:37.629 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.629+0000 7f202bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:32:37.629 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.629+0000 7f202bfff640 10 monclient(hunting): dump: 2024-03-29T11:32:37.629 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:37.629 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:37.629 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:37.629 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:37.629 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:37.629 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:37.629 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:37.629 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:37.629 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:37.630 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:37.630 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.629+0000 7f202bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:37.630 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.629+0000 7f202bfff640 1 -- 172.21.15.135:0/3307403794 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114697450 0 0) 0x7f204413ec10 con 0x7f2044149dc0 2024-03-29T11:32:37.630 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.629+0000 7f202bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:37.630 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.629+0000 7f202bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:37.630 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.629+0000 7f202bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:37.630 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.629+0000 7f202bfff640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:32:37.630 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.629+0000 7f202bfff640 1 -- 172.21.15.135:0/3307403794 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2030005850 con 0x7f2044149dc0 2024-03-29T11:32:37.630 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.630+0000 7f202bfff640 1 -- 172.21.15.135:0/3307403794 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2895914581 0 0) 0x7f2030005300 con 0x7f20440a9930 2024-03-29T11:32:37.630 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.630+0000 7f202bfff640 1 -- 172.21.15.135:0/3307403794 >> v1:172.21.15.135:6790/0 conn(0x7f204414ff40 legacy=0x7f204413fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:37.630 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.630+0000 7f202bfff640 1 -- 172.21.15.135:0/3307403794 >> v1:172.21.15.194:6789/0 conn(0x7f2044149dc0 legacy=0x7f204413f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:37.630 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.630+0000 7f202bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:32:37.630 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.630+0000 7f202bfff640 1 monclient: found mon.a 2024-03-29T11:32:37.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.630+0000 7f202bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:37.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.630+0000 7f202bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:32:37.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.630+0000 7f202bfff640 1 -- 172.21.15.135:0/3307403794 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2044149140 con 0x7f20440a9930 2024-03-29T11:32:37.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.630+0000 7f202bfff640 10 monclient: _finish_auth 0 2024-03-29T11:32:37.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.630+0000 7f202bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:32:37.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.630+0000 7f202bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:37.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.630+0000 7f2049863640 5 monclient: authenticate success, global_id 4503 2024-03-29T11:32:37.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.630+0000 7f2049863640 10 monclient: _renew_subs 2024-03-29T11:32:37.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.630+0000 7f2049863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:32:37.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.630+0000 7f2049863640 1 -- 172.21.15.135:0/3307403794 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20441389d0 con 0x7f20440a9930 2024-03-29T11:32:37.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.630+0000 7f2049863640 10 client.4503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:37.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.630+0000 7f202bfff640 1 -- 172.21.15.135:0/3307403794 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2034000ed0 con 0x7f20440a9930 2024-03-29T11:32:37.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.630+0000 7f202bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:37.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.630+0000 7f202bfff640 1 -- 172.21.15.135:0/3307403794 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f2034004040 con 0x7f20440a9930 2024-03-29T11:32:37.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.630+0000 7f202bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:37.632 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.630+0000 7f202bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:37.632 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.630+0000 7f202bfff640 10 monclient: dump: 2024-03-29T11:32:37.632 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:37.632 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:37.632 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:37.632 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:37.632 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:37.632 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:37.632 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:37.632 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:37.632 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:37.632 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:37.632 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.630+0000 7f20417fa640 4 set_mon_vals no callback set 2024-03-29T11:32:37.632 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.632+0000 7f2049863640 10 monclient: _renew_subs 2024-03-29T11:32:37.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.632+0000 7f2049863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:32:37.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.632+0000 7f2049863640 1 -- 172.21.15.135:0/3307403794 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2044148bc0 con 0x7f20440a9930 2024-03-29T11:32:37.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.632+0000 7f2049863640 1 librados: init done 2024-03-29T11:32:37.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.633+0000 7f202bfff640 1 -- 172.21.15.135:0/3307403794 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f2034001180 con 0x7f20440a9930 2024-03-29T11:32:37.634 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.634+0000 7f2049863640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:37.634 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.634+0000 7f2049863640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:37.634 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.634+0000 7f2049863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:32:37.634 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.634+0000 7f2049863640 1 -- 172.21.15.135:0/3307403794 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2044075f10 con 0x7f20440a9930 2024-03-29T11:32:37.634 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.635+0000 7f20427fc640 10 monclient: get_auth_request con 0x7f203006e270 auth_method 0 2024-03-29T11:32:37.634 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.635+0000 7f202bfff640 1 -- 172.21.15.135:0/3307403794 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f2034054600 con 0x7f20440a9930 2024-03-29T11:32:37.634 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.635+0000 7f202bfff640 10 client.4503.objecter ms_dispatch 0x7f2044000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:37.634 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.635+0000 7f202bfff640 3 client.4503.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:37.634 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.635+0000 7f202bfff640 3 client.4503.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:37.634 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.635+0000 7f202bfff640 20 client.4503.objecter dump_active .. 0 homeless 2024-03-29T11:32:37.634 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.635+0000 7f202bfff640 10 client.4503.objecter ms_handle_connect 0x7f203006e270 2024-03-29T11:32:37.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.642+0000 7f202bfff640 1 -- 172.21.15.135:0/3307403794 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2034050110 con 0x7f20440a9930 2024-03-29T11:32:37.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.642+0000 7f202bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:37.641 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.642+0000 7f202bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:37.959 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.964+0000 7f2049863640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:37.959 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.964+0000 7f2049863640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:37.959 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.964+0000 7f2049863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:32:37.959 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.964+0000 7f2049863640 1 -- 172.21.15.135:0/3307403794 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20441492d0 con 0x7f20440a9930 2024-03-29T11:32:37.961 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.967+0000 7f202bfff640 1 -- 172.21.15.135:0/3307403794 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f2034050110 con 0x7f20440a9930 2024-03-29T11:32:37.962 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.967+0000 7f202bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:37.962 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.967+0000 7f202bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:37.962 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:32:37.962 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:37.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.970+0000 7f2049863640 10 librados: watch_flush enter 2024-03-29T11:32:37.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.970+0000 7f2049863640 10 librados: watch_flush exit 2024-03-29T11:32:37.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.970+0000 7f2049863640 20 client.4503.objecter shutdown clearing up homeless session... 2024-03-29T11:32:37.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.970+0000 7f2049863640 10 client.4503.objecter successfully canceled tick 2024-03-29T11:32:37.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.971+0000 7f2049863640 1 -- 172.21.15.135:0/3307403794 >> v1:172.21.15.135:6816/3469875285 conn(0x7f203006e270 legacy=0x7f2030070730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:37.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.971+0000 7f2049863640 10 monclient: shutdown 2024-03-29T11:32:37.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.971+0000 7f2049863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:37.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.971+0000 7f2049863640 1 -- 172.21.15.135:0/3307403794 >> v1:172.21.15.135:6789/0 conn(0x7f20440a9930 legacy=0x7f20441382c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:37.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.971+0000 7f2049863640 1 -- 172.21.15.135:0/3307403794 shutdown_connections 2024-03-29T11:32:37.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.971+0000 7f2049863640 1 -- 172.21.15.135:0/3307403794 >> 172.21.15.135:0/3307403794 conn(0x7f20440b35e0 msgr2=0x7f20440b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:37.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.974+0000 7f2049863640 1 -- 172.21.15.135:0/3307403794 shutdown_connections 2024-03-29T11:32:37.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.974+0000 7f2049863640 1 -- 172.21.15.135:0/3307403794 wait complete. 2024-03-29T11:32:37.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:37.974+0000 7f2049863640 1 librados: shutdown 2024-03-29T11:32:37.988 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:44.795 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:44.943 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.946+0000 7f2e77fff640 10 monclient: get_monmap_and_config 2024-03-29T11:32:44.943 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.946+0000 7f2e77fff640 10 monclient: build_initial_monmap 2024-03-29T11:32:44.943 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.946+0000 7f2e77fff640 10 monclient: monmap: 2024-03-29T11:32:44.943 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:32:44.943 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:44.943 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:32:44.948544+0000 2024-03-29T11:32:44.943 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:32:44.948544+0000 2024-03-29T11:32:44.943 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:44.944 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:44.944 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:32:44.944 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:32:44.944 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:32:44.944 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:44.944 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.948+0000 7f2e77fff640 1 Processor -- start 2024-03-29T11:32:44.945 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.948+0000 7f2e77fff640 1 -- start start 2024-03-29T11:32:44.945 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.948+0000 7f2e77fff640 10 monclient: init 2024-03-29T11:32:44.945 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.949+0000 7f2e77fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:44.945 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.949+0000 7f2e77fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:32:44.945 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.949+0000 7f2e77fff640 10 monclient(hunting): picked mon.noname-a con 0x7f2e70038c60 addr v1:172.21.15.135:6789/0 2024-03-29T11:32:44.945 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.949+0000 7f2e77fff640 10 monclient(hunting): picked mon.noname-b con 0x7f2e700324b0 addr v1:172.21.15.135:6790/0 2024-03-29T11:32:44.946 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.949+0000 7f2e77fff640 10 monclient(hunting): picked mon.noname-c con 0x7f2e700d11e0 addr v1:172.21.15.194:6789/0 2024-03-29T11:32:44.946 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.949+0000 7f2e77fff640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e700caf80 con 0x7f2e70038c60 2024-03-29T11:32:44.946 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.949+0000 7f2e77fff640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e70039850 con 0x7f2e700d11e0 2024-03-29T11:32:44.946 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.949+0000 7f2e77fff640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e700c78d0 con 0x7f2e700324b0 2024-03-29T11:32:44.946 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.949+0000 7f2e77fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:44.946 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.949+0000 7f2e77fff640 10 monclient(hunting): authenticate will time out at 1272.318585s 2024-03-29T11:32:44.946 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.949+0000 7f2e767fc640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f2e700324b0 0x7f2e700cb4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:33016/0 (socket says 172.21.15.135:33016) 2024-03-29T11:32:44.946 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.949+0000 7f2e767fc640 1 -- 172.21.15.135:0/2704945525 learned_addr learned my addr 172.21.15.135:0/2704945525 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:32:44.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 1 -- 172.21.15.135:0/2704945525 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1535259429 0 0) 0x7f2e700c78d0 con 0x7f2e700324b0 2024-03-29T11:32:44.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:44.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:44.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:44.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:32:44.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 1 -- 172.21.15.135:0/2704945525 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e64003520 con 0x7f2e700324b0 2024-03-29T11:32:44.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 1 -- 172.21.15.135:0/2704945525 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126519255 0 0) 0x7f2e70039850 con 0x7f2e700d11e0 2024-03-29T11:32:44.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:44.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:44.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:44.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:32:44.947 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 1 -- 172.21.15.135:0/2704945525 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e700c78d0 con 0x7f2e700d11e0 2024-03-29T11:32:44.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 1 -- 172.21.15.135:0/2704945525 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 30618541 0 0) 0x7f2e64003520 con 0x7f2e700324b0 2024-03-29T11:32:44.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 1 -- 172.21.15.135:0/2704945525 >> v1:172.21.15.194:6789/0 conn(0x7f2e700d11e0 legacy=0x7f2e700d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:44.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 1 -- 172.21.15.135:0/2704945525 >> v1:172.21.15.135:6789/0 conn(0x7f2e70038c60 legacy=0x7f2e700c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:44.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:32:44.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 1 monclient: found mon.noname-b 2024-03-29T11:32:44.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:44.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:32:44.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 1 -- 172.21.15.135:0/2704945525 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e700c7a60 con 0x7f2e700324b0 2024-03-29T11:32:44.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 10 monclient: _finish_auth 0 2024-03-29T11:32:44.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:32:44.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:44.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 1 -- 172.21.15.135:0/2704945525 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f2e60001690 con 0x7f2e700324b0 2024-03-29T11:32:44.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:44.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:44.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e75ffb640 10 monclient: dump: 2024-03-29T11:32:44.949 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:44.949 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:44.949 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:44.949 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:44.949 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:44.949 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:44.949 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:44.949 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:44.949 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:44.949 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:44.949 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e77fff640 5 monclient: authenticate success, global_id 4541 2024-03-29T11:32:44.949 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.950+0000 7f2e77fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:44.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.951+0000 7f2e75ffb640 1 -- 172.21.15.135:0/2704945525 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e600017f0 con 0x7f2e700324b0 2024-03-29T11:32:44.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.951+0000 7f2e75ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:44.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.951+0000 7f2e77fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:44.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.951+0000 7f2e77fff640 4 set_mon_vals no callback set 2024-03-29T11:32:44.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.952+0000 7f2e77fff640 10 monclient: shutdown 2024-03-29T11:32:44.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.952+0000 7f2e77fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:44.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.952+0000 7f2e77fff640 1 -- 172.21.15.135:0/2704945525 >> v1:172.21.15.135:6790/0 conn(0x7f2e700324b0 legacy=0x7f2e700cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:44.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.952+0000 7f2e77fff640 1 -- 172.21.15.135:0/2704945525 shutdown_connections 2024-03-29T11:32:44.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.952+0000 7f2e77fff640 1 -- 172.21.15.135:0/2704945525 >> 172.21.15.135:0/2704945525 conn(0x7f2e70005c20 msgr2=0x7f2e70006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:44.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.953+0000 7f2e77fff640 1 -- 172.21.15.135:0/2704945525 shutdown_connections 2024-03-29T11:32:44.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.953+0000 7f2e77fff640 1 -- 172.21.15.135:0/2704945525 wait complete. 2024-03-29T11:32:44.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.955+0000 7f2e77fff640 10 monclient: build_initial_monmap 2024-03-29T11:32:44.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.955+0000 7f2e77fff640 10 monclient: monmap: 2024-03-29T11:32:44.952 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:32:44.952 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:44.952 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:32:44.952 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:32:44.952 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:44.952 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:44.952 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:32:44.952 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:32:44.952 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:32:44.952 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:44.954 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.955+0000 7f2e77fff640 1 librados: starting msgr at 2024-03-29T11:32:44.954 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.955+0000 7f2e77fff640 1 librados: starting objecter 2024-03-29T11:32:44.954 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.956+0000 7f2e77fff640 1 Processor -- start 2024-03-29T11:32:44.954 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.956+0000 7f2e77fff640 1 -- start start 2024-03-29T11:32:44.954 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.956+0000 7f2e77fff640 1 librados: setting wanted keys 2024-03-29T11:32:44.954 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.956+0000 7f2e77fff640 1 librados: calling monclient init 2024-03-29T11:32:44.954 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.956+0000 7f2e77fff640 10 monclient: init 2024-03-29T11:32:44.954 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.957+0000 7f2e77fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:44.954 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.957+0000 7f2e77fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:32:44.954 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.957+0000 7f2e77fff640 10 monclient(hunting): picked mon.noname-c con 0x7f2e700324b0 addr v1:172.21.15.135:6790/0 2024-03-29T11:32:44.955 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.957+0000 7f2e77fff640 10 monclient(hunting): picked mon.noname-a con 0x7f2e70038c60 addr v1:172.21.15.135:6789/0 2024-03-29T11:32:44.955 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.957+0000 7f2e77fff640 10 monclient(hunting): picked mon.noname-b con 0x7f2e700d11e0 addr v1:172.21.15.194:6789/0 2024-03-29T11:32:44.955 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.957+0000 7f2e77fff640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e700d3ae0 con 0x7f2e70038c60 2024-03-29T11:32:44.955 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.957+0000 7f2e77fff640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e701648a0 con 0x7f2e700d11e0 2024-03-29T11:32:44.955 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.957+0000 7f2e77fff640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e70164410 con 0x7f2e700324b0 2024-03-29T11:32:44.955 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.957+0000 7f2e77fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:44.955 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.957+0000 7f2e77fff640 10 monclient(hunting): authenticate will time out at 1272.326728s 2024-03-29T11:32:44.955 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.957+0000 7f2e767fc640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f2e70038c60 0x7f2e70034af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:59282/0 (socket says 172.21.15.135:59282) 2024-03-29T11:32:44.955 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.957+0000 7f2e767fc640 1 -- 172.21.15.135:0/1531480452 learned_addr learned my addr 172.21.15.135:0/1531480452 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:32:44.955 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.958+0000 7f2e6f7fe640 10 client.?.objecter ms_handle_connect 0x7f2e70038c60 2024-03-29T11:32:44.955 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.958+0000 7f2e6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:44.955 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.958+0000 7f2e6f7fe640 10 client.?.objecter ms_handle_connect 0x7f2e700d11e0 2024-03-29T11:32:44.955 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.958+0000 7f2e6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:44.957 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.958+0000 7f2e6f7fe640 1 -- 172.21.15.135:0/1531480452 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 590098618 0 0) 0x7f2e700d3ae0 con 0x7f2e70038c60 2024-03-29T11:32:44.957 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.958+0000 7f2e6f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:44.957 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.958+0000 7f2e6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:44.957 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.958+0000 7f2e6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:44.957 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.958+0000 7f2e6f7fe640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:32:44.957 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.958+0000 7f2e6f7fe640 1 -- 172.21.15.135:0/1531480452 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e58004eb0 con 0x7f2e70038c60 2024-03-29T11:32:44.957 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.958+0000 7f2e6f7fe640 10 client.?.objecter ms_handle_connect 0x7f2e700324b0 2024-03-29T11:32:44.958 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.958+0000 7f2e6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:44.958 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.959+0000 7f2e6f7fe640 1 -- 172.21.15.135:0/1531480452 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222832295 0 0) 0x7f2e701648a0 con 0x7f2e700d11e0 2024-03-29T11:32:44.958 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.959+0000 7f2e6f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:44.958 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.959+0000 7f2e6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:44.958 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.959+0000 7f2e6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:44.958 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.959+0000 7f2e6f7fe640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:32:44.958 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.959+0000 7f2e6f7fe640 1 -- 172.21.15.135:0/1531480452 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e700d3ae0 con 0x7f2e700d11e0 2024-03-29T11:32:44.958 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.959+0000 7f2e6f7fe640 1 -- 172.21.15.135:0/1531480452 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483807754 0 0) 0x7f2e70164410 con 0x7f2e700324b0 2024-03-29T11:32:44.958 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.959+0000 7f2e6f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:32:44.958 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.959+0000 7f2e6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:44.958 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.959+0000 7f2e6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:44.958 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.959+0000 7f2e6f7fe640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:32:44.958 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.959+0000 7f2e6f7fe640 1 -- 172.21.15.135:0/1531480452 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e701648a0 con 0x7f2e700324b0 2024-03-29T11:32:44.959 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.959+0000 7f2e6f7fe640 1 -- 172.21.15.135:0/1531480452 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1074202468 0 0) 0x7f2e700d3ae0 con 0x7f2e700d11e0 2024-03-29T11:32:44.959 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.959+0000 7f2e6f7fe640 1 -- 172.21.15.135:0/1531480452 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e70164410 con 0x7f2e700d11e0 2024-03-29T11:32:44.959 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.959+0000 7f2e6f7fe640 1 -- 172.21.15.135:0/1531480452 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f2e680017d0 con 0x7f2e700d11e0 2024-03-29T11:32:44.959 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.959+0000 7f2e6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:44.959 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.959+0000 7f2e6f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:44.959 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.959+0000 7f2e6f7fe640 10 monclient(hunting): dump: 2024-03-29T11:32:44.959 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:44.959 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:44.959 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:44.959 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:44.959 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:44.959 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:44.959 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:44.959 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:44.960 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:44.960 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:44.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.959+0000 7f2e6f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:44.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.959+0000 7f2e6f7fe640 1 -- 172.21.15.135:0/1531480452 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 663602736 0 0) 0x7f2e58004eb0 con 0x7f2e70038c60 2024-03-29T11:32:44.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.960+0000 7f2e6f7fe640 1 -- 172.21.15.135:0/1531480452 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e700d3ae0 con 0x7f2e70038c60 2024-03-29T11:32:44.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.960+0000 7f2e6f7fe640 1 -- 172.21.15.135:0/1531480452 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f2e60001ca0 con 0x7f2e70038c60 2024-03-29T11:32:44.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.960+0000 7f2e6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:44.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.960+0000 7f2e6f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:44.960 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.960+0000 7f2e6f7fe640 10 monclient(hunting): dump: 2024-03-29T11:32:44.960 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:44.961 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:44.961 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:44.961 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:44.961 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:44.961 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:44.961 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:44.961 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:44.961 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:44.961 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:44.961 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.960+0000 7f2e6f7fe640 1 -- 172.21.15.135:0/1531480452 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3065361460 0 0) 0x7f2e70164410 con 0x7f2e700d11e0 2024-03-29T11:32:44.961 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.960+0000 7f2e6f7fe640 1 -- 172.21.15.135:0/1531480452 >> v1:172.21.15.135:6790/0 conn(0x7f2e700324b0 legacy=0x7f2e70164dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:44.961 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.960+0000 7f2e6f7fe640 1 -- 172.21.15.135:0/1531480452 >> v1:172.21.15.135:6789/0 conn(0x7f2e70038c60 legacy=0x7f2e70034af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:44.961 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.960+0000 7f2e6f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:32:44.961 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.960+0000 7f2e6f7fe640 1 monclient: found mon.b 2024-03-29T11:32:44.961 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.960+0000 7f2e6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:44.962 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.960+0000 7f2e6f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:32:44.962 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.960+0000 7f2e6f7fe640 1 -- 172.21.15.135:0/1531480452 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e701645a0 con 0x7f2e700d11e0 2024-03-29T11:32:44.962 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.960+0000 7f2e6f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:32:44.962 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.960+0000 7f2e6f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:32:44.962 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.960+0000 7f2e6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:44.962 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.960+0000 7f2e77fff640 5 monclient: authenticate success, global_id 4522 2024-03-29T11:32:44.962 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.960+0000 7f2e77fff640 10 monclient: _renew_subs 2024-03-29T11:32:44.962 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.960+0000 7f2e77fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:32:44.962 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.960+0000 7f2e77fff640 1 -- 172.21.15.135:0/1531480452 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e700b88a0 con 0x7f2e700d11e0 2024-03-29T11:32:44.962 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.960+0000 7f2e77fff640 10 client.4522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:44.962 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.960+0000 7f2e77fff640 10 monclient: _renew_subs 2024-03-29T11:32:44.962 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.960+0000 7f2e77fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:32:44.962 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.960+0000 7f2e77fff640 1 -- 172.21.15.135:0/1531480452 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e700d4270 con 0x7f2e700d11e0 2024-03-29T11:32:44.962 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.960+0000 7f2e77fff640 1 librados: init done 2024-03-29T11:32:44.963 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.961+0000 7f2e6f7fe640 1 -- 172.21.15.135:0/1531480452 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e68001930 con 0x7f2e700d11e0 2024-03-29T11:32:44.963 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.961+0000 7f2e6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:44.963 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.961+0000 7f2e6f7fe640 1 -- 172.21.15.135:0/1531480452 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f2e680037b0 con 0x7f2e700d11e0 2024-03-29T11:32:44.963 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.961+0000 7f2e6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:44.963 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.961+0000 7f2e6f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:44.963 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.961+0000 7f2e6f7fe640 10 monclient: dump: 2024-03-29T11:32:44.963 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:44.963 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:44.963 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:44.963 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:44.963 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:44.963 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:44.963 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:44.963 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:44.963 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:44.964 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:44.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.961+0000 7f2e757fa640 4 set_mon_vals no callback set 2024-03-29T11:32:44.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.962+0000 7f2e77fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:44.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.963+0000 7f2e6f7fe640 1 -- 172.21.15.135:0/1531480452 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f2e68019780 con 0x7f2e700d11e0 2024-03-29T11:32:44.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.963+0000 7f2e6f7fe640 1 -- 172.21.15.135:0/1531480452 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f2e680837a0 con 0x7f2e700d11e0 2024-03-29T11:32:44.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.963+0000 7f2e6f7fe640 10 client.4522.objecter ms_dispatch 0x7f2e78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:44.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.963+0000 7f2e6f7fe640 3 client.4522.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:44.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.963+0000 7f2e6f7fe640 3 client.4522.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:44.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.963+0000 7f2e77fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:44.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.963+0000 7f2e77fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:32:44.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.963+0000 7f2e6f7fe640 20 client.4522.objecter dump_active .. 0 homeless 2024-03-29T11:32:44.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.963+0000 7f2e77fff640 1 -- 172.21.15.135:0/1531480452 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e5c005080 con 0x7f2e700d11e0 2024-03-29T11:32:44.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.964+0000 7f2e76ffd640 10 monclient: get_auth_request con 0x7f2e5806de80 auth_method 0 2024-03-29T11:32:44.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.964+0000 7f2e6f7fe640 10 client.4522.objecter ms_handle_connect 0x7f2e5806de80 2024-03-29T11:32:44.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.972+0000 7f2e6f7fe640 1 -- 172.21.15.135:0/1531480452 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2e6804eeb0 con 0x7f2e700d11e0 2024-03-29T11:32:44.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.972+0000 7f2e6f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:44.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:44.972+0000 7f2e6f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:45.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:45.341+0000 7f2e77fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:45.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:45.341+0000 7f2e77fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:45.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:45.341+0000 7f2e77fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:32:45.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:45.341+0000 7f2e77fff640 1 -- 172.21.15.135:0/1531480452 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e5c005330 con 0x7f2e700d11e0 2024-03-29T11:32:45.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:45.342+0000 7f2e6f7fe640 1 -- 172.21.15.135:0/1531480452 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f2e68050fb0 con 0x7f2e700d11e0 2024-03-29T11:32:45.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:45.342+0000 7f2e6f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:45.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:45.342+0000 7f2e6f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:45.339 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:32:45.339 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:45.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:45.345+0000 7f2e77fff640 10 librados: watch_flush enter 2024-03-29T11:32:45.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:45.345+0000 7f2e77fff640 10 librados: watch_flush exit 2024-03-29T11:32:45.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:45.345+0000 7f2e77fff640 20 client.4522.objecter shutdown clearing up homeless session... 2024-03-29T11:32:45.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:45.345+0000 7f2e77fff640 10 client.4522.objecter successfully canceled tick 2024-03-29T11:32:45.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:45.345+0000 7f2e77fff640 1 -- 172.21.15.135:0/1531480452 >> v1:172.21.15.135:6816/3469875285 conn(0x7f2e5806de80 legacy=0x7f2e58070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:45.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:45.345+0000 7f2e77fff640 10 monclient: shutdown 2024-03-29T11:32:45.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:45.345+0000 7f2e77fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:45.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:45.345+0000 7f2e77fff640 1 -- 172.21.15.135:0/1531480452 >> v1:172.21.15.194:6789/0 conn(0x7f2e700d11e0 legacy=0x7f2e700b8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:45.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:45.345+0000 7f2e77fff640 1 -- 172.21.15.135:0/1531480452 shutdown_connections 2024-03-29T11:32:45.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:45.345+0000 7f2e77fff640 1 -- 172.21.15.135:0/1531480452 >> 172.21.15.135:0/1531480452 conn(0x7f2e70005c20 msgr2=0x7f2e7003bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:45.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:45.346+0000 7f2e77fff640 1 -- 172.21.15.135:0/1531480452 shutdown_connections 2024-03-29T11:32:45.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:45.346+0000 7f2e77fff640 1 -- 172.21.15.135:0/1531480452 wait complete. 2024-03-29T11:32:45.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:45.346+0000 7f2e77fff640 1 librados: shutdown 2024-03-29T11:32:45.358 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:52.165 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:52.314 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.316+0000 7fb57bfff640 10 monclient: get_monmap_and_config 2024-03-29T11:32:52.314 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.316+0000 7fb57bfff640 10 monclient: build_initial_monmap 2024-03-29T11:32:52.314 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.316+0000 7fb57bfff640 10 monclient: monmap: 2024-03-29T11:32:52.314 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:32:52.314 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:52.315 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:32:52.318430+0000 2024-03-29T11:32:52.315 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:32:52.318430+0000 2024-03-29T11:32:52.315 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:52.315 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:52.315 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:32:52.315 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:32:52.315 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:32:52.315 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:52.319 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.319+0000 7fb57bfff640 1 Processor -- start 2024-03-29T11:32:52.319 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.320+0000 7fb57bfff640 1 -- start start 2024-03-29T11:32:52.319 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.320+0000 7fb57bfff640 10 monclient: init 2024-03-29T11:32:52.319 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.320+0000 7fb57bfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:52.319 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.320+0000 7fb57bfff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:32:52.319 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.320+0000 7fb57bfff640 10 monclient(hunting): picked mon.noname-c con 0x7fb57c0a9750 addr v1:172.21.15.194:6789/0 2024-03-29T11:32:52.319 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.320+0000 7fb57bfff640 10 monclient(hunting): picked mon.noname-b con 0x7fb57c0adca0 addr v1:172.21.15.135:6790/0 2024-03-29T11:32:52.319 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.320+0000 7fb57bfff640 10 monclient(hunting): picked mon.noname-a con 0x7fb57c146a80 addr v1:172.21.15.135:6789/0 2024-03-29T11:32:52.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.320+0000 7fb57bfff640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57c0abdc0 con 0x7fb57c146a80 2024-03-29T11:32:52.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.320+0000 7fb57bfff640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57c0b0070 con 0x7fb57c0a9750 2024-03-29T11:32:52.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.320+0000 7fb57bfff640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57c0af880 con 0x7fb57c0adca0 2024-03-29T11:32:52.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.320+0000 7fb57bfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:52.320 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.320+0000 7fb57bfff640 10 monclient(hunting): authenticate will time out at 1279.690029s 2024-03-29T11:32:52.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.324+0000 7fb57b7fe640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fb57c146a80 0x7fb57c151bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:57725/0 (socket says 172.21.15.135:57725) 2024-03-29T11:32:52.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.324+0000 7fb57b7fe640 1 -- 172.21.15.135:0/3850978936 learned_addr learned my addr 172.21.15.135:0/3850978936 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:32:52.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.324+0000 7fb579ffb640 1 -- 172.21.15.135:0/3850978936 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136301401 0 0) 0x7fb57c0abdc0 con 0x7fb57c146a80 2024-03-29T11:32:52.322 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.324+0000 7fb579ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:52.324 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.324+0000 7fb579ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:52.324 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.324+0000 7fb579ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:52.324 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.324+0000 7fb579ffb640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:32:52.324 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.324+0000 7fb579ffb640 1 -- 172.21.15.135:0/3850978936 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb564003520 con 0x7fb57c146a80 2024-03-29T11:32:52.324 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.324+0000 7fb579ffb640 1 -- 172.21.15.135:0/3850978936 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1566278811 0 0) 0x7fb564003520 con 0x7fb57c146a80 2024-03-29T11:32:52.324 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.324+0000 7fb579ffb640 1 -- 172.21.15.135:0/3850978936 >> v1:172.21.15.135:6790/0 conn(0x7fb57c0adca0 legacy=0x7fb57c146370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:32:52.324 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.325+0000 7fb579ffb640 1 -- 172.21.15.135:0/3850978936 >> v1:172.21.15.194:6789/0 conn(0x7fb57c0a9750 legacy=0x7fb57c143710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:32:52.324 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.326+0000 7fb579ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:32:52.324 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.326+0000 7fb579ffb640 1 monclient: found mon.noname-a 2024-03-29T11:32:52.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.326+0000 7fb579ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:52.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.326+0000 7fb579ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.135:6789/0 2024-03-29T11:32:52.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.326+0000 7fb579ffb640 1 -- 172.21.15.135:0/3850978936 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57c0ac030 con 0x7fb57c146a80 2024-03-29T11:32:52.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.326+0000 7fb579ffb640 10 monclient: _finish_auth 0 2024-03-29T11:32:52.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.326+0000 7fb579ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:32:52.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.326+0000 7fb579ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:52.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.326+0000 7fb579ffb640 1 -- 172.21.15.135:0/3850978936 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fb56c001670 con 0x7fb57c146a80 2024-03-29T11:32:52.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.326+0000 7fb579ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:52.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.326+0000 7fb579ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:32:52.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.326+0000 7fb579ffb640 10 monclient: dump: 2024-03-29T11:32:52.325 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:52.325 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:52.325 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:52.325 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:52.325 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:52.326 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:52.326 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:52.326 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:52.326 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:52.326 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:52.327 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.328+0000 7fb57bfff640 5 monclient: authenticate success, global_id 4512 2024-03-29T11:32:52.327 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.328+0000 7fb57bfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:52.327 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.330+0000 7fb579ffb640 1 -- 172.21.15.135:0/3850978936 <== mon.0 v1:172.21.15.135:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb56c0018d0 con 0x7fb57c146a80 2024-03-29T11:32:52.327 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.330+0000 7fb579ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:52.328 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.330+0000 7fb579ffb640 1 -- 172.21.15.135:0/3850978936 <== mon.0 v1:172.21.15.135:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fb56c001d20 con 0x7fb57c146a80 2024-03-29T11:32:52.329 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.330+0000 7fb579ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:52.329 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.330+0000 7fb579ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:52.329 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.330+0000 7fb579ffb640 10 monclient: dump: 2024-03-29T11:32:52.329 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:52.329 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:52.329 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:52.329 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:52.329 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:52.329 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:52.329 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:52.329 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:52.329 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:52.329 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:52.329 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.330+0000 7fb57bfff640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:52.330 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.330+0000 7fb57bfff640 4 set_mon_vals no callback set 2024-03-29T11:32:52.330 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.331+0000 7fb57bfff640 10 monclient: shutdown 2024-03-29T11:32:52.330 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.331+0000 7fb57bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:52.330 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.331+0000 7fb57bfff640 1 -- 172.21.15.135:0/3850978936 >> v1:172.21.15.135:6789/0 conn(0x7fb57c146a80 legacy=0x7fb57c151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:52.330 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.331+0000 7fb57bfff640 1 -- 172.21.15.135:0/3850978936 shutdown_connections 2024-03-29T11:32:52.330 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.331+0000 7fb57bfff640 1 -- 172.21.15.135:0/3850978936 >> 172.21.15.135:0/3850978936 conn(0x7fb57c0673e0 msgr2=0x7fb57c0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:52.330 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.331+0000 7fb57bfff640 1 -- 172.21.15.135:0/3850978936 shutdown_connections 2024-03-29T11:32:52.330 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.331+0000 7fb57bfff640 1 -- 172.21.15.135:0/3850978936 wait complete. 2024-03-29T11:32:52.331 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.333+0000 7fb57bfff640 10 monclient: build_initial_monmap 2024-03-29T11:32:52.331 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.333+0000 7fb57bfff640 10 monclient: monmap: 2024-03-29T11:32:52.331 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:32:52.331 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:52.331 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:32:52.331 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:32:52.331 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:52.331 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:52.331 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:32:52.331 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:32:52.331 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:32:52.331 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:52.334 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.333+0000 7fb57bfff640 1 librados: starting msgr at 2024-03-29T11:32:52.334 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.333+0000 7fb57bfff640 1 librados: starting objecter 2024-03-29T11:32:52.334 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.333+0000 7fb57bfff640 1 Processor -- start 2024-03-29T11:32:52.334 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.334+0000 7fb57bfff640 1 -- start start 2024-03-29T11:32:52.334 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.334+0000 7fb57bfff640 1 librados: setting wanted keys 2024-03-29T11:32:52.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.334+0000 7fb57bfff640 1 librados: calling monclient init 2024-03-29T11:32:52.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.334+0000 7fb57bfff640 10 monclient: init 2024-03-29T11:32:52.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.334+0000 7fb57bfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:52.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.334+0000 7fb57bfff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:32:52.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.334+0000 7fb57bfff640 10 monclient(hunting): picked mon.noname-a con 0x7fb57c0a9750 addr v1:172.21.15.135:6789/0 2024-03-29T11:32:52.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.334+0000 7fb57bfff640 10 monclient(hunting): picked mon.noname-c con 0x7fb57c0adca0 addr v1:172.21.15.135:6790/0 2024-03-29T11:32:52.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.334+0000 7fb57bfff640 10 monclient(hunting): picked mon.noname-b con 0x7fb57c146a80 addr v1:172.21.15.194:6789/0 2024-03-29T11:32:52.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.334+0000 7fb57bfff640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57c1dead0 con 0x7fb57c0a9750 2024-03-29T11:32:52.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.334+0000 7fb57bfff640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57c1ded20 con 0x7fb57c146a80 2024-03-29T11:32:52.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.334+0000 7fb57bfff640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57c1df950 con 0x7fb57c0adca0 2024-03-29T11:32:52.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.334+0000 7fb57bfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:52.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.334+0000 7fb57bfff640 10 monclient(hunting): authenticate will time out at 1279.703995s 2024-03-29T11:32:52.337 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.338+0000 7fb57affd640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fb57c0a9750 0x7fb57c1df240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:57955/0 (socket says 172.21.15.135:57955) 2024-03-29T11:32:52.337 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.338+0000 7fb57affd640 1 -- 172.21.15.135:0/3871992302 learned_addr learned my addr 172.21.15.135:0/3871992302 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:32:52.337 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.339+0000 7fb563fff640 10 client.?.objecter ms_handle_connect 0x7fb57c0a9750 2024-03-29T11:32:52.337 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.339+0000 7fb563fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:52.337 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.339+0000 7fb563fff640 1 -- 172.21.15.135:0/3871992302 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439416085 0 0) 0x7fb57c1dead0 con 0x7fb57c0a9750 2024-03-29T11:32:52.337 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.339+0000 7fb563fff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:52.337 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.339+0000 7fb563fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:52.337 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.339+0000 7fb563fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:52.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.339+0000 7fb563fff640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:32:52.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.339+0000 7fb563fff640 1 -- 172.21.15.135:0/3871992302 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb568004eb0 con 0x7fb57c0a9750 2024-03-29T11:32:52.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.340+0000 7fb563fff640 1 -- 172.21.15.135:0/3871992302 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1265697443 0 0) 0x7fb568004eb0 con 0x7fb57c0a9750 2024-03-29T11:32:52.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.340+0000 7fb563fff640 1 -- 172.21.15.135:0/3871992302 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb57c1dead0 con 0x7fb57c0a9750 2024-03-29T11:32:52.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.340+0000 7fb563fff640 1 -- 172.21.15.135:0/3871992302 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fb570001870 con 0x7fb57c0a9750 2024-03-29T11:32:52.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.340+0000 7fb563fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:52.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.340+0000 7fb563fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:32:52.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.340+0000 7fb563fff640 10 monclient(hunting): dump: 2024-03-29T11:32:52.339 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:52.339 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:52.339 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:52.340 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:52.340 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:52.340 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:52.340 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:52.340 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:52.340 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:52.340 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:52.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.340+0000 7fb563fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:52.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.340+0000 7fb563fff640 10 client.?.objecter ms_handle_connect 0x7fb57c0adca0 2024-03-29T11:32:52.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.340+0000 7fb563fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:52.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.340+0000 7fb563fff640 10 client.?.objecter ms_handle_connect 0x7fb57c146a80 2024-03-29T11:32:52.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.340+0000 7fb563fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:52.345 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.340+0000 7fb563fff640 1 -- 172.21.15.135:0/3871992302 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761805401 0 0) 0x7fb57c1df950 con 0x7fb57c0adca0 2024-03-29T11:32:52.345 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.340+0000 7fb563fff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:52.345 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.340+0000 7fb563fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:52.345 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.340+0000 7fb563fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:52.345 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.340+0000 7fb563fff640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:32:52.345 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.340+0000 7fb563fff640 1 -- 172.21.15.135:0/3871992302 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb568004eb0 con 0x7fb57c0adca0 2024-03-29T11:32:52.345 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.342+0000 7fb563fff640 1 -- 172.21.15.135:0/3871992302 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 416113647 0 0) 0x7fb568004eb0 con 0x7fb57c0adca0 2024-03-29T11:32:52.345 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.342+0000 7fb563fff640 1 -- 172.21.15.135:0/3871992302 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb57c1df950 con 0x7fb57c0adca0 2024-03-29T11:32:52.346 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.342+0000 7fb563fff640 1 -- 172.21.15.135:0/3871992302 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fb574001b10 con 0x7fb57c0adca0 2024-03-29T11:32:52.346 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.342+0000 7fb563fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:52.346 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.342+0000 7fb563fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:52.346 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.342+0000 7fb563fff640 10 monclient(hunting): dump: 2024-03-29T11:32:52.346 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:52.346 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:52.346 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:52.346 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:52.346 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:52.346 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:52.346 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:52.346 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:52.346 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:52.346 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:52.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.342+0000 7fb563fff640 1 -- 172.21.15.135:0/3871992302 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4032867495 0 0) 0x7fb57c1dead0 con 0x7fb57c0a9750 2024-03-29T11:32:52.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.343+0000 7fb563fff640 1 -- 172.21.15.135:0/3871992302 >> v1:172.21.15.135:6790/0 conn(0x7fb57c0adca0 legacy=0x7fb57c1db800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:52.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.343+0000 7fb563fff640 1 -- 172.21.15.135:0/3871992302 >> v1:172.21.15.194:6789/0 conn(0x7fb57c146a80 legacy=0x7fb57c1dbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:52.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.343+0000 7fb563fff640 10 monclient: _finish_hunting 0 2024-03-29T11:32:52.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.343+0000 7fb563fff640 1 monclient: found mon.a 2024-03-29T11:32:52.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.343+0000 7fb563fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:52.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.343+0000 7fb563fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:32:52.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.343+0000 7fb563fff640 1 -- 172.21.15.135:0/3871992302 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57c0b1410 con 0x7fb57c0a9750 2024-03-29T11:32:52.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.343+0000 7fb563fff640 10 monclient: _finish_auth 0 2024-03-29T11:32:52.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.343+0000 7fb563fff640 10 monclient: _check_auth_tickets 2024-03-29T11:32:52.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.343+0000 7fb563fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:52.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.343+0000 7fb57bfff640 5 monclient: authenticate success, global_id 4515 2024-03-29T11:32:52.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.343+0000 7fb57bfff640 10 monclient: _renew_subs 2024-03-29T11:32:52.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.343+0000 7fb57bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:32:52.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.343+0000 7fb57bfff640 1 -- 172.21.15.135:0/3871992302 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb57c0b16c0 con 0x7fb57c0a9750 2024-03-29T11:32:52.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.343+0000 7fb57bfff640 10 client.4515.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:52.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.343+0000 7fb57bfff640 10 monclient: _renew_subs 2024-03-29T11:32:52.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.343+0000 7fb57bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:32:52.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.343+0000 7fb57bfff640 1 -- 172.21.15.135:0/3871992302 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb57c1e7240 con 0x7fb57c0a9750 2024-03-29T11:32:52.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.343+0000 7fb57bfff640 1 librados: init done 2024-03-29T11:32:52.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.345+0000 7fb563fff640 1 -- 172.21.15.135:0/3871992302 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5700019d0 con 0x7fb57c0a9750 2024-03-29T11:32:52.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.345+0000 7fb563fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:52.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.345+0000 7fb563fff640 1 -- 172.21.15.135:0/3871992302 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fb570004ae0 con 0x7fb57c0a9750 2024-03-29T11:32:52.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.345+0000 7fb563fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:52.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.345+0000 7fb563fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:32:52.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.345+0000 7fb563fff640 10 monclient: dump: 2024-03-29T11:32:52.348 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:52.348 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:52.348 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:52.349 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:52.349 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:52.349 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:52.349 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:52.349 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:52.349 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:52.349 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:52.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.346+0000 7fb57bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:52.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.346+0000 7fb57bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:52.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.346+0000 7fb57bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:32:52.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.346+0000 7fb57bfff640 1 -- 172.21.15.135:0/3871992302 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb57c04f730 con 0x7fb57c0a9750 2024-03-29T11:32:52.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.346+0000 7fb563fff640 1 -- 172.21.15.135:0/3871992302 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fb57001ab70 con 0x7fb57c0a9750 2024-03-29T11:32:52.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.346+0000 7fb578ff9640 4 set_mon_vals no callback set 2024-03-29T11:32:52.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.347+0000 7fb563fff640 1 -- 172.21.15.135:0/3871992302 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7fb570084b30 con 0x7fb57c0a9750 2024-03-29T11:32:52.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.347+0000 7fb563fff640 10 client.4515.objecter ms_dispatch 0x7fb57c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:52.350 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.347+0000 7fb563fff640 3 client.4515.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:52.350 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.347+0000 7fb563fff640 3 client.4515.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:52.350 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.347+0000 7fb563fff640 20 client.4515.objecter dump_active .. 0 homeless 2024-03-29T11:32:52.350 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.347+0000 7fb57a7fc640 10 monclient: get_auth_request con 0x7fb56806e000 auth_method 0 2024-03-29T11:32:52.350 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.348+0000 7fb563fff640 10 client.4515.objecter ms_handle_connect 0x7fb56806e000 2024-03-29T11:32:52.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.355+0000 7fb563fff640 1 -- 172.21.15.135:0/3871992302 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb570050240 con 0x7fb57c0a9750 2024-03-29T11:32:52.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.355+0000 7fb563fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:52.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.355+0000 7fb563fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:52.684 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.687+0000 7fb57bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:52.684 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.687+0000 7fb57bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:52.684 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.687+0000 7fb57bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:32:52.684 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.687+0000 7fb57bfff640 1 -- 172.21.15.135:0/3871992302 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb57c0ab110 con 0x7fb57c0a9750 2024-03-29T11:32:52.684 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.688+0000 7fb563fff640 1 -- 172.21.15.135:0/3871992302 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7fb570052340 con 0x7fb57c0a9750 2024-03-29T11:32:52.685 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.688+0000 7fb563fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:52.685 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.688+0000 7fb563fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:52.685 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:32:52.685 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:52.688 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.692+0000 7fb57bfff640 10 librados: watch_flush enter 2024-03-29T11:32:52.688 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.692+0000 7fb57bfff640 10 librados: watch_flush exit 2024-03-29T11:32:52.688 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.692+0000 7fb57bfff640 20 client.4515.objecter shutdown clearing up homeless session... 2024-03-29T11:32:52.688 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.692+0000 7fb57bfff640 10 client.4515.objecter successfully canceled tick 2024-03-29T11:32:52.688 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.692+0000 7fb57bfff640 1 -- 172.21.15.135:0/3871992302 >> v1:172.21.15.135:6816/3469875285 conn(0x7fb56806e000 legacy=0x7fb5680704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:52.688 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.692+0000 7fb57bfff640 10 monclient: shutdown 2024-03-29T11:32:52.689 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.692+0000 7fb57bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:52.689 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.692+0000 7fb57bfff640 1 -- 172.21.15.135:0/3871992302 >> v1:172.21.15.135:6789/0 conn(0x7fb57c0a9750 legacy=0x7fb57c1df240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:52.689 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.692+0000 7fb57bfff640 1 -- 172.21.15.135:0/3871992302 shutdown_connections 2024-03-29T11:32:52.689 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.692+0000 7fb57bfff640 1 -- 172.21.15.135:0/3871992302 >> 172.21.15.135:0/3871992302 conn(0x7fb57c0673e0 msgr2=0x7fb57c0aa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:52.689 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.692+0000 7fb57bfff640 1 -- 172.21.15.135:0/3871992302 shutdown_connections 2024-03-29T11:32:52.689 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.692+0000 7fb57bfff640 1 -- 172.21.15.135:0/3871992302 wait complete. 2024-03-29T11:32:52.689 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:52.692+0000 7fb57bfff640 1 librados: shutdown 2024-03-29T11:32:52.704 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:32:59.511 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:32:59.645 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.649+0000 7fb85f68f640 10 monclient: get_monmap_and_config 2024-03-29T11:32:59.645 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.649+0000 7fb85f68f640 10 monclient: build_initial_monmap 2024-03-29T11:32:59.646 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.649+0000 7fb85f68f640 10 monclient: monmap: 2024-03-29T11:32:59.646 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:32:59.646 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:59.646 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:32:59.650916+0000 2024-03-29T11:32:59.646 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:32:59.650916+0000 2024-03-29T11:32:59.646 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:59.646 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:59.646 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:32:59.646 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:32:59.646 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:32:59.646 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:59.649 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.650+0000 7fb85f68f640 1 Processor -- start 2024-03-29T11:32:59.649 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.650+0000 7fb85f68f640 1 -- start start 2024-03-29T11:32:59.649 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.650+0000 7fb85f68f640 10 monclient: init 2024-03-29T11:32:59.649 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.650+0000 7fb85f68f640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:59.649 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.650+0000 7fb85f68f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:32:59.649 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.650+0000 7fb85f68f640 10 monclient(hunting): picked mon.noname-b con 0x7fb8581457f0 addr v1:172.21.15.135:6790/0 2024-03-29T11:32:59.649 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.650+0000 7fb85f68f640 10 monclient(hunting): picked mon.noname-a con 0x7fb85813f650 addr v1:172.21.15.135:6789/0 2024-03-29T11:32:59.649 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.650+0000 7fb85f68f640 10 monclient(hunting): picked mon.noname-c con 0x7fb8580a9d70 addr v1:172.21.15.194:6789/0 2024-03-29T11:32:59.649 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.650+0000 7fb85f68f640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8580aaef0 con 0x7fb85813f650 2024-03-29T11:32:59.649 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.650+0000 7fb85f68f640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb858145bd0 con 0x7fb8580a9d70 2024-03-29T11:32:59.649 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.651+0000 7fb85f68f640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb858145eb0 con 0x7fb8581457f0 2024-03-29T11:32:59.650 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.651+0000 7fb85f68f640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:59.650 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.651+0000 7fb85f68f640 10 monclient(hunting): authenticate will time out at 1287.020182s 2024-03-29T11:32:59.650 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.651+0000 7fb85dc2c640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fb8580a9d70 0x7fb85814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:39404/0 (socket says 172.21.15.135:39404) 2024-03-29T11:32:59.650 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.651+0000 7fb85dc2c640 1 -- 172.21.15.135:0/85313080 learned_addr learned my addr 172.21.15.135:0/85313080 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:32:59.650 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.651+0000 7fb84ffff640 1 -- 172.21.15.135:0/85313080 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618114621 0 0) 0x7fb858145bd0 con 0x7fb8580a9d70 2024-03-29T11:32:59.650 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.651+0000 7fb84ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:32:59.650 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.651+0000 7fb84ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:59.650 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.651+0000 7fb84ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:59.650 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.651+0000 7fb84ffff640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:32:59.650 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.651+0000 7fb84ffff640 1 -- 172.21.15.135:0/85313080 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb844003520 con 0x7fb8580a9d70 2024-03-29T11:32:59.650 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.651+0000 7fb84ffff640 1 -- 172.21.15.135:0/85313080 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 789617343 0 0) 0x7fb844003520 con 0x7fb8580a9d70 2024-03-29T11:32:59.650 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.651+0000 7fb84ffff640 1 -- 172.21.15.135:0/85313080 >> v1:172.21.15.135:6790/0 conn(0x7fb8581457f0 legacy=0x7fb85813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:59.650 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.651+0000 7fb84ffff640 1 -- 172.21.15.135:0/85313080 >> v1:172.21.15.135:6789/0 conn(0x7fb85813f650 legacy=0x7fb8580b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:32:59.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.654+0000 7fb84ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:32:59.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.654+0000 7fb84ffff640 1 monclient: found mon.noname-c 2024-03-29T11:32:59.651 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.654+0000 7fb84ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:59.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.654+0000 7fb84ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:32:59.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.654+0000 7fb84ffff640 1 -- 172.21.15.135:0/85313080 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb858146340 con 0x7fb8580a9d70 2024-03-29T11:32:59.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.654+0000 7fb84ffff640 10 monclient: _finish_auth 0 2024-03-29T11:32:59.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.654+0000 7fb84ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:32:59.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.654+0000 7fb84ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:59.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.654+0000 7fb84ffff640 1 -- 172.21.15.135:0/85313080 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fb850001a10 con 0x7fb8580a9d70 2024-03-29T11:32:59.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.654+0000 7fb84ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:59.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.654+0000 7fb84ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:32:59.652 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.654+0000 7fb84ffff640 10 monclient: dump: 2024-03-29T11:32:59.652 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:59.652 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:59.652 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:59.652 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:59.652 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:59.653 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:59.653 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:59.653 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:59.653 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:59.653 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:59.653 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.654+0000 7fb85f68f640 5 monclient: authenticate success, global_id 4528 2024-03-29T11:32:59.653 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.654+0000 7fb85f68f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:32:59.653 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.654+0000 7fb84ffff640 1 -- 172.21.15.135:0/85313080 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb850001ca0 con 0x7fb8580a9d70 2024-03-29T11:32:59.653 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.654+0000 7fb84ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:59.653 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.654+0000 7fb84ffff640 1 -- 172.21.15.135:0/85313080 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fb8500020b0 con 0x7fb8580a9d70 2024-03-29T11:32:59.653 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.654+0000 7fb84ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:59.653 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.654+0000 7fb84ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:59.653 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.654+0000 7fb84ffff640 10 monclient: dump: 2024-03-29T11:32:59.653 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:59.653 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:59.654 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:59.654 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:59.654 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:59.654 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:59.654 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:59.654 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:59.654 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:59.654 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:59.654 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.654+0000 7fb85f68f640 10 monclient: get_monmap_and_config success 2024-03-29T11:32:59.654 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.654+0000 7fb85f68f640 4 set_mon_vals no callback set 2024-03-29T11:32:59.656 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.655+0000 7fb85f68f640 10 monclient: shutdown 2024-03-29T11:32:59.656 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.655+0000 7fb85f68f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:59.656 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.655+0000 7fb85f68f640 1 -- 172.21.15.135:0/85313080 >> v1:172.21.15.194:6789/0 conn(0x7fb8580a9d70 legacy=0x7fb85814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:59.656 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.656+0000 7fb85f68f640 1 -- 172.21.15.135:0/85313080 shutdown_connections 2024-03-29T11:32:59.656 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.656+0000 7fb85f68f640 1 -- 172.21.15.135:0/85313080 >> 172.21.15.135:0/85313080 conn(0x7fb858066a00 msgr2=0x7fb858066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:59.656 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.656+0000 7fb85f68f640 1 -- 172.21.15.135:0/85313080 shutdown_connections 2024-03-29T11:32:59.656 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.656+0000 7fb85f68f640 1 -- 172.21.15.135:0/85313080 wait complete. 2024-03-29T11:32:59.656 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.657+0000 7fb85f68f640 10 monclient: build_initial_monmap 2024-03-29T11:32:59.656 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.657+0000 7fb85f68f640 10 monclient: monmap: 2024-03-29T11:32:59.656 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:32:59.656 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:32:59.656 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:32:59.657 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:32:59.657 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:32:59.657 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:59.657 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:32:59.657 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:32:59.657 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:32:59.657 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:59.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.657+0000 7fb85f68f640 1 librados: starting msgr at 2024-03-29T11:32:59.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.657+0000 7fb85f68f640 1 librados: starting objecter 2024-03-29T11:32:59.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.657+0000 7fb85f68f640 1 Processor -- start 2024-03-29T11:32:59.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.658+0000 7fb85f68f640 1 -- start start 2024-03-29T11:32:59.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.658+0000 7fb85f68f640 1 librados: setting wanted keys 2024-03-29T11:32:59.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.658+0000 7fb85f68f640 1 librados: calling monclient init 2024-03-29T11:32:59.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.658+0000 7fb85f68f640 10 monclient: init 2024-03-29T11:32:59.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.658+0000 7fb85f68f640 10 monclient: _reopen_session rank -1 2024-03-29T11:32:59.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.658+0000 7fb85f68f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:32:59.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.658+0000 7fb85f68f640 10 monclient(hunting): picked mon.noname-b con 0x7fb8580a9d70 addr v1:172.21.15.194:6789/0 2024-03-29T11:32:59.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.658+0000 7fb85f68f640 10 monclient(hunting): picked mon.noname-c con 0x7fb85813f650 addr v1:172.21.15.135:6790/0 2024-03-29T11:32:59.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.658+0000 7fb85d42b640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fb8580a9d70 0x7fb85813fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:40050/0 (socket says 172.21.15.135:40050) 2024-03-29T11:32:59.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.659+0000 7fb85cc2a640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7fb85813f650 0x7fb858136da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:48410/0 (socket says 172.21.15.135:48410) 2024-03-29T11:32:59.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.660+0000 7fb85f68f640 10 monclient(hunting): picked mon.noname-a con 0x7fb8581457f0 addr v1:172.21.15.135:6789/0 2024-03-29T11:32:59.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.660+0000 7fb85f68f640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb858137bc0 con 0x7fb8581457f0 2024-03-29T11:32:59.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.660+0000 7fb85f68f640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8580ac7f0 con 0x7fb8580a9d70 2024-03-29T11:32:59.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.660+0000 7fb85f68f640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb858141e10 con 0x7fb85813f650 2024-03-29T11:32:59.658 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.660+0000 7fb85f68f640 10 monclient(hunting): _renew_subs 2024-03-29T11:32:59.659 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.660+0000 7fb85f68f640 10 monclient(hunting): authenticate will time out at 1287.030011s 2024-03-29T11:32:59.659 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.662+0000 7fb85d42b640 1 -- 172.21.15.135:0/3222090376 learned_addr learned my addr 172.21.15.135:0/3222090376 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:32:59.659 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.662+0000 7fb84dffb640 10 client.?.objecter ms_handle_connect 0x7fb85813f650 2024-03-29T11:32:59.659 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.662+0000 7fb84dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:59.659 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.663+0000 7fb84dffb640 10 client.?.objecter ms_handle_connect 0x7fb8581457f0 2024-03-29T11:32:59.659 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.663+0000 7fb84dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:59.663 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.663+0000 7fb84dffb640 10 client.?.objecter ms_handle_connect 0x7fb8580a9d70 2024-03-29T11:32:59.663 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.663+0000 7fb84dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:32:59.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.663+0000 7fb84dffb640 1 -- 172.21.15.135:0/3222090376 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1623190011 0 0) 0x7fb858137bc0 con 0x7fb8581457f0 2024-03-29T11:32:59.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.663+0000 7fb84dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:59.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.663+0000 7fb84dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:59.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.663+0000 7fb84dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:59.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.663+0000 7fb84dffb640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:32:59.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.664+0000 7fb84dffb640 1 -- 172.21.15.135:0/3222090376 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb83c004eb0 con 0x7fb8581457f0 2024-03-29T11:32:59.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.664+0000 7fb84dffb640 1 -- 172.21.15.135:0/3222090376 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2081697050 0 0) 0x7fb8580ac7f0 con 0x7fb8580a9d70 2024-03-29T11:32:59.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.664+0000 7fb84dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:59.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.664+0000 7fb84dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:59.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.664+0000 7fb84dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:59.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.664+0000 7fb84dffb640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:32:59.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.664+0000 7fb84dffb640 1 -- 172.21.15.135:0/3222090376 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb858137bc0 con 0x7fb8580a9d70 2024-03-29T11:32:59.664 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.664+0000 7fb84dffb640 1 -- 172.21.15.135:0/3222090376 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 503080559 0 0) 0x7fb858141e10 con 0x7fb85813f650 2024-03-29T11:32:59.665 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.664+0000 7fb84dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:32:59.665 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.664+0000 7fb84dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:32:59.665 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.664+0000 7fb84dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:32:59.665 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.664+0000 7fb84dffb640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:32:59.665 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.664+0000 7fb84dffb640 1 -- 172.21.15.135:0/3222090376 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8580ac7f0 con 0x7fb85813f650 2024-03-29T11:32:59.665 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.664+0000 7fb84dffb640 1 -- 172.21.15.135:0/3222090376 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2202180930 0 0) 0x7fb858137bc0 con 0x7fb8580a9d70 2024-03-29T11:32:59.665 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.664+0000 7fb84dffb640 1 -- 172.21.15.135:0/3222090376 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb858141e10 con 0x7fb8580a9d70 2024-03-29T11:32:59.665 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.664+0000 7fb84dffb640 1 -- 172.21.15.135:0/3222090376 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fb854001850 con 0x7fb8580a9d70 2024-03-29T11:32:59.665 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.664+0000 7fb84dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:59.665 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.664+0000 7fb84dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:32:59.665 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.664+0000 7fb84dffb640 10 monclient(hunting): dump: 2024-03-29T11:32:59.665 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:59.665 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:59.665 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:59.665 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:59.665 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:59.666 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:59.666 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:59.666 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:59.666 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:59.666 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:59.666 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.664+0000 7fb84dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:32:59.666 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.664+0000 7fb84dffb640 1 -- 172.21.15.135:0/3222090376 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4072933909 0 0) 0x7fb8580ac7f0 con 0x7fb85813f650 2024-03-29T11:32:59.666 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb84dffb640 1 -- 172.21.15.135:0/3222090376 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb858137bc0 con 0x7fb85813f650 2024-03-29T11:32:59.666 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb84dffb640 1 -- 172.21.15.135:0/3222090376 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fb848001760 con 0x7fb85813f650 2024-03-29T11:32:59.666 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb84dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:59.666 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb84dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:32:59.666 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb84dffb640 10 monclient(hunting): dump: 2024-03-29T11:32:59.666 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:59.666 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:59.666 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:59.667 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:59.667 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:59.667 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:59.667 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:59.667 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:59.667 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:59.667 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:59.667 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb84dffb640 1 -- 172.21.15.135:0/3222090376 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2615131115 0 0) 0x7fb858141e10 con 0x7fb8580a9d70 2024-03-29T11:32:59.667 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb84dffb640 1 -- 172.21.15.135:0/3222090376 >> v1:172.21.15.135:6790/0 conn(0x7fb85813f650 legacy=0x7fb858136da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:59.667 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb84dffb640 1 -- 172.21.15.135:0/3222090376 >> v1:172.21.15.135:6789/0 conn(0x7fb8581457f0 legacy=0x7fb8581374b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:59.667 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb84dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:32:59.667 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb84dffb640 1 monclient: found mon.b 2024-03-29T11:32:59.667 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb84dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:32:59.667 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb84dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:32:59.667 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb84dffb640 1 -- 172.21.15.135:0/3222090376 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8580b17b0 con 0x7fb8580a9d70 2024-03-29T11:32:59.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb84dffb640 10 monclient: _finish_auth 0 2024-03-29T11:32:59.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb84dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:32:59.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb84dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:32:59.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb85f68f640 5 monclient: authenticate success, global_id 4531 2024-03-29T11:32:59.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb85f68f640 10 monclient: _renew_subs 2024-03-29T11:32:59.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb85f68f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:32:59.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb85f68f640 1 -- 172.21.15.135:0/3222090376 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb858142d20 con 0x7fb8580a9d70 2024-03-29T11:32:59.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb84dffb640 1 -- 172.21.15.135:0/3222090376 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8540019b0 con 0x7fb8580a9d70 2024-03-29T11:32:59.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb84dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:32:59.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb84f7fe640 4 set_mon_vals no callback set 2024-03-29T11:32:59.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb85f68f640 10 client.4531.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:32:59.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb85f68f640 10 monclient: _renew_subs 2024-03-29T11:32:59.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb85f68f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:32:59.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb85f68f640 1 -- 172.21.15.135:0/3222090376 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8580ab7e0 con 0x7fb8580a9d70 2024-03-29T11:32:59.668 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.665+0000 7fb85f68f640 1 librados: init done 2024-03-29T11:32:59.669 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.666+0000 7fb84dffb640 1 -- 172.21.15.135:0/3222090376 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fb8540037b0 con 0x7fb8580a9d70 2024-03-29T11:32:59.669 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.666+0000 7fb84dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:32:59.669 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.666+0000 7fb84dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:32:59.669 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.666+0000 7fb84dffb640 10 monclient: dump: 2024-03-29T11:32:59.669 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:32:59.669 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:32:59.669 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:59.669 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:32:59.669 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:32:59.669 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:32:59.669 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:32:59.669 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:32:59.669 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:32:59.669 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:32:59.669 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.667+0000 7fb84dffb640 1 -- 172.21.15.135:0/3222090376 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fb8540039d0 con 0x7fb8580a9d70 2024-03-29T11:32:59.670 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.668+0000 7fb84dffb640 1 -- 172.21.15.135:0/3222090376 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7fb854084910 con 0x7fb8580a9d70 2024-03-29T11:32:59.671 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.668+0000 7fb84dffb640 10 client.4531.objecter ms_dispatch 0x7fb858000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:32:59.671 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.668+0000 7fb84dffb640 3 client.4531.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:32:59.671 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.668+0000 7fb84dffb640 3 client.4531.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:32:59.671 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.668+0000 7fb84dffb640 20 client.4531.objecter dump_active .. 0 homeless 2024-03-29T11:32:59.671 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.669+0000 7fb85cc2a640 10 monclient: get_auth_request con 0x7fb83c06dee0 auth_method 0 2024-03-29T11:32:59.671 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.669+0000 7fb85f68f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:59.671 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.669+0000 7fb85f68f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:59.671 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.669+0000 7fb85f68f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:32:59.671 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.669+0000 7fb85f68f640 1 -- 172.21.15.135:0/3222090376 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8581429a0 con 0x7fb8580a9d70 2024-03-29T11:32:59.671 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.670+0000 7fb84dffb640 10 client.4531.objecter ms_handle_connect 0x7fb83c06dee0 2024-03-29T11:32:59.674 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.678+0000 7fb84dffb640 1 -- 172.21.15.135:0/3222090376 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb854050250 con 0x7fb8580a9d70 2024-03-29T11:32:59.674 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.678+0000 7fb84dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:32:59.674 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.678+0000 7fb84dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:32:59.987 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.990+0000 7fb85f68f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:59.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.991+0000 7fb85f68f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:59.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.991+0000 7fb85f68f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:32:59.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.991+0000 7fb85f68f640 1 -- 172.21.15.135:0/3222090376 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb858075f10 con 0x7fb8580a9d70 2024-03-29T11:32:59.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.992+0000 7fb84dffb640 1 -- 172.21.15.135:0/3222090376 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7fb854083fc0 con 0x7fb8580a9d70 2024-03-29T11:32:59.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.992+0000 7fb84dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:32:59.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.992+0000 7fb84dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:32:59.989 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:32:59.989 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:32:59.994 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.996+0000 7fb85f68f640 10 librados: watch_flush enter 2024-03-29T11:32:59.994 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.996+0000 7fb85f68f640 10 librados: watch_flush exit 2024-03-29T11:32:59.995 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.996+0000 7fb85f68f640 20 client.4531.objecter shutdown clearing up homeless session... 2024-03-29T11:32:59.995 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.996+0000 7fb85f68f640 10 client.4531.objecter successfully canceled tick 2024-03-29T11:32:59.995 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.996+0000 7fb85f68f640 1 -- 172.21.15.135:0/3222090376 >> v1:172.21.15.135:6816/3469875285 conn(0x7fb83c06dee0 legacy=0x7fb83c0703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:59.995 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.996+0000 7fb85f68f640 10 monclient: shutdown 2024-03-29T11:32:59.995 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.996+0000 7fb85f68f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:32:59.995 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.996+0000 7fb85f68f640 1 -- 172.21.15.135:0/3222090376 >> v1:172.21.15.194:6789/0 conn(0x7fb8580a9d70 legacy=0x7fb85813fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:32:59.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.999+0000 7fb85f68f640 1 -- 172.21.15.135:0/3222090376 shutdown_connections 2024-03-29T11:32:59.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.999+0000 7fb85f68f640 1 -- 172.21.15.135:0/3222090376 >> 172.21.15.135:0/3222090376 conn(0x7fb858066a00 msgr2=0x7fb858066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:32:59.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.999+0000 7fb85f68f640 1 -- 172.21.15.135:0/3222090376 shutdown_connections 2024-03-29T11:32:59.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.999+0000 7fb85f68f640 1 -- 172.21.15.135:0/3222090376 wait complete. 2024-03-29T11:32:59.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:32:59.999+0000 7fb85f68f640 1 librados: shutdown 2024-03-29T11:33:00.013 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:06.820 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:06.972 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.977+0000 7f2dbe7af640 10 monclient: get_monmap_and_config 2024-03-29T11:33:06.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.977+0000 7f2dbe7af640 10 monclient: build_initial_monmap 2024-03-29T11:33:06.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.977+0000 7f2dbe7af640 10 monclient: monmap: 2024-03-29T11:33:06.973 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:33:06.973 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:06.973 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:33:06.977964+0000 2024-03-29T11:33:06.973 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:33:06.977964+0000 2024-03-29T11:33:06.973 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:06.973 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:06.973 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:33:06.973 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:33:06.973 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:33:06.973 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:06.976 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.979+0000 7f2dbe7af640 1 Processor -- start 2024-03-29T11:33:06.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.979+0000 7f2dbe7af640 1 -- start start 2024-03-29T11:33:06.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.979+0000 7f2dbe7af640 10 monclient: init 2024-03-29T11:33:06.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.979+0000 7f2dbe7af640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:06.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.979+0000 7f2dbe7af640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:33:06.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.980+0000 7f2dbe7af640 10 monclient(hunting): picked mon.noname-a con 0x7f2db80b4330 addr v1:172.21.15.135:6789/0 2024-03-29T11:33:06.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.980+0000 7f2dbe7af640 10 monclient(hunting): picked mon.noname-b con 0x7f2db81470b0 addr v1:172.21.15.135:6790/0 2024-03-29T11:33:06.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.980+0000 7f2dbe7af640 10 monclient(hunting): picked mon.noname-c con 0x7f2db81439b0 addr v1:172.21.15.194:6789/0 2024-03-29T11:33:06.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.981+0000 7f2dbe7af640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db806f2a0 con 0x7f2db80b4330 2024-03-29T11:33:06.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.981+0000 7f2db7fff640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f2db80b4330 0x7f2db80b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:34996/0 (socket says 172.21.15.135:34996) 2024-03-29T11:33:06.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.981+0000 7f2db7fff640 1 -- 172.21.15.135:0/3973651100 learned_addr learned my addr 172.21.15.135:0/3973651100 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:33:06.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.981+0000 7f2dbe7af640 1 -- 172.21.15.135:0/3973651100 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db8146b90 con 0x7f2db81439b0 2024-03-29T11:33:06.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.981+0000 7f2dbe7af640 1 -- 172.21.15.135:0/3973651100 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db80b2ca0 con 0x7f2db81470b0 2024-03-29T11:33:06.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.981+0000 7f2dbe7af640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:06.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.981+0000 7f2dbe7af640 10 monclient(hunting): authenticate will time out at 1294.349960s 2024-03-29T11:33:06.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.982+0000 7f2db6ffd640 1 -- 172.21.15.135:0/3973651100 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59867271 0 0) 0x7f2db806f2a0 con 0x7f2db80b4330 2024-03-29T11:33:06.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.982+0000 7f2db6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:33:06.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.982+0000 7f2db6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:06.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.982+0000 7f2db6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:06.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.982+0000 7f2db6ffd640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:33:06.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.982+0000 7f2db6ffd640 1 -- 172.21.15.135:0/3973651100 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2da4003520 con 0x7f2db80b4330 2024-03-29T11:33:06.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.982+0000 7f2db6ffd640 1 -- 172.21.15.135:0/3973651100 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1769965322 0 0) 0x7f2da4003520 con 0x7f2db80b4330 2024-03-29T11:33:06.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.982+0000 7f2db6ffd640 1 -- 172.21.15.135:0/3973651100 >> v1:172.21.15.135:6790/0 conn(0x7f2db81470b0 legacy=0x7f2db80b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:33:06.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.984+0000 7f2db6ffd640 1 -- 172.21.15.135:0/3973651100 >> v1:172.21.15.194:6789/0 conn(0x7f2db81439b0 legacy=0x7f2db8151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:06.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.984+0000 7f2db6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:33:06.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.984+0000 7f2db6ffd640 1 monclient: found mon.noname-a 2024-03-29T11:33:06.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.984+0000 7f2db6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:06.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.984+0000 7f2db6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.135:6789/0 2024-03-29T11:33:06.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.984+0000 7f2db6ffd640 1 -- 172.21.15.135:0/3973651100 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db806e0c0 con 0x7f2db80b4330 2024-03-29T11:33:06.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.984+0000 7f2db6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:33:06.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.984+0000 7f2db6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:33:06.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.984+0000 7f2db6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:06.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.984+0000 7f2db6ffd640 1 -- 172.21.15.135:0/3973651100 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f2db0001a10 con 0x7f2db80b4330 2024-03-29T11:33:06.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.984+0000 7f2dbe7af640 5 monclient: authenticate success, global_id 4521 2024-03-29T11:33:06.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.984+0000 7f2dbe7af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:06.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.984+0000 7f2db6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:06.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.985+0000 7f2db6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:33:06.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.985+0000 7f2db6ffd640 10 monclient: dump: 2024-03-29T11:33:06.983 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:06.983 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:06.983 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:06.983 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:06.983 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:06.983 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:06.983 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:06.983 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:06.984 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:06.984 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:06.984 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.985+0000 7f2db6ffd640 1 -- 172.21.15.135:0/3973651100 <== mon.0 v1:172.21.15.135:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2db0001ca0 con 0x7f2db80b4330 2024-03-29T11:33:06.984 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.985+0000 7f2dbe7af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:06.984 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.985+0000 7f2db6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:06.984 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.985+0000 7f2db6ffd640 1 -- 172.21.15.135:0/3973651100 <== mon.0 v1:172.21.15.135:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f2db0002030 con 0x7f2db80b4330 2024-03-29T11:33:06.984 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.985+0000 7f2dbe7af640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:06.984 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.985+0000 7f2dbe7af640 4 set_mon_vals no callback set 2024-03-29T11:33:06.984 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.986+0000 7f2dbe7af640 10 monclient: shutdown 2024-03-29T11:33:06.984 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.986+0000 7f2dbe7af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:06.984 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.986+0000 7f2dbe7af640 1 -- 172.21.15.135:0/3973651100 >> v1:172.21.15.135:6789/0 conn(0x7f2db80b4330 legacy=0x7f2db80b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:06.984 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.986+0000 7f2db6ffd640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:33:06.984 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.986+0000 7f2dbe7af640 1 -- 172.21.15.135:0/3973651100 shutdown_connections 2024-03-29T11:33:06.984 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.986+0000 7f2dbe7af640 1 -- 172.21.15.135:0/3973651100 >> 172.21.15.135:0/3973651100 conn(0x7f2db8066a00 msgr2=0x7f2db8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:06.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.987+0000 7f2dbe7af640 1 -- 172.21.15.135:0/3973651100 shutdown_connections 2024-03-29T11:33:06.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.987+0000 7f2dbe7af640 1 -- 172.21.15.135:0/3973651100 wait complete. 2024-03-29T11:33:06.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.988+0000 7f2dbe7af640 10 monclient: build_initial_monmap 2024-03-29T11:33:06.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.988+0000 7f2dbe7af640 10 monclient: monmap: 2024-03-29T11:33:06.986 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:33:06.986 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:06.986 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:33:06.986 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:33:06.986 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:06.986 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:06.986 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:33:06.986 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:33:06.986 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:33:06.986 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:06.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.988+0000 7f2dbe7af640 1 librados: starting msgr at 2024-03-29T11:33:06.987 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.988+0000 7f2dbe7af640 1 librados: starting objecter 2024-03-29T11:33:06.987 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.988+0000 7f2dbe7af640 1 Processor -- start 2024-03-29T11:33:06.987 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.989+0000 7f2dbe7af640 1 -- start start 2024-03-29T11:33:06.987 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.989+0000 7f2dbe7af640 1 librados: setting wanted keys 2024-03-29T11:33:06.987 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.989+0000 7f2dbe7af640 1 librados: calling monclient init 2024-03-29T11:33:06.987 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.989+0000 7f2dbe7af640 10 monclient: init 2024-03-29T11:33:06.987 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.990+0000 7f2dbe7af640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:06.987 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.990+0000 7f2dbe7af640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:33:06.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.991+0000 7f2dbe7af640 10 monclient(hunting): picked mon.noname-c con 0x7f2db80b4330 addr v1:172.21.15.135:6790/0 2024-03-29T11:33:06.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.991+0000 7f2dbe7af640 10 monclient(hunting): picked mon.noname-b con 0x7f2db81439b0 addr v1:172.21.15.194:6789/0 2024-03-29T11:33:06.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.991+0000 7f2dbe7af640 10 monclient(hunting): picked mon.noname-a con 0x7f2db81470b0 addr v1:172.21.15.135:6789/0 2024-03-29T11:33:06.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.991+0000 7f2dbe7af640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db81219c0 con 0x7f2db81470b0 2024-03-29T11:33:06.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.991+0000 7f2dbe7af640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db81dc250 con 0x7f2db81439b0 2024-03-29T11:33:06.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.991+0000 7f2dbe7af640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db81dbf70 con 0x7f2db80b4330 2024-03-29T11:33:06.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.991+0000 7f2dbe7af640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:06.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.991+0000 7f2dbe7af640 10 monclient(hunting): authenticate will time out at 1294.359139s 2024-03-29T11:33:06.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.992+0000 7f2db7fff640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f2db80b4330 0x7f2db81db860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:38335/0 (socket says 172.21.15.135:38335) 2024-03-29T11:33:06.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.992+0000 7f2db7fff640 1 -- 172.21.15.135:0/2981836170 learned_addr learned my addr 172.21.15.135:0/2981836170 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:33:06.993 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.993+0000 7f2db4ff9640 10 client.?.objecter ms_handle_connect 0x7f2db80b4330 2024-03-29T11:33:06.993 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.993+0000 7f2db4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:06.993 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.993+0000 7f2db4ff9640 1 -- 172.21.15.135:0/2981836170 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440974017 0 0) 0x7f2db81dbf70 con 0x7f2db80b4330 2024-03-29T11:33:06.993 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.993+0000 7f2db4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:33:06.993 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.993+0000 7f2db4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:06.993 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.993+0000 7f2db4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:06.993 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.993+0000 7f2db4ff9640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:33:06.994 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.993+0000 7f2db4ff9640 1 -- 172.21.15.135:0/2981836170 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d98004eb0 con 0x7f2db80b4330 2024-03-29T11:33:06.994 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.993+0000 7f2db4ff9640 10 client.?.objecter ms_handle_connect 0x7f2db81470b0 2024-03-29T11:33:06.994 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.993+0000 7f2db4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:06.994 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.993+0000 7f2db4ff9640 1 -- 172.21.15.135:0/2981836170 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392873824 0 0) 0x7f2db81219c0 con 0x7f2db81470b0 2024-03-29T11:33:06.994 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.993+0000 7f2db4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:33:06.994 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.994+0000 7f2db4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:06.994 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.994+0000 7f2db4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:06.994 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.994+0000 7f2db4ff9640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:33:06.994 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.994+0000 7f2db4ff9640 1 -- 172.21.15.135:0/2981836170 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2db81dbf70 con 0x7f2db81470b0 2024-03-29T11:33:06.994 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.994+0000 7f2db4ff9640 1 -- 172.21.15.135:0/2981836170 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 707603669 0 0) 0x7f2d98004eb0 con 0x7f2db80b4330 2024-03-29T11:33:06.994 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.994+0000 7f2db4ff9640 1 -- 172.21.15.135:0/2981836170 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2db81219c0 con 0x7f2db80b4330 2024-03-29T11:33:06.994 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.994+0000 7f2db4ff9640 1 -- 172.21.15.135:0/2981836170 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f2db0001a10 con 0x7f2db80b4330 2024-03-29T11:33:06.994 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.994+0000 7f2db4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:06.995 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.994+0000 7f2db4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:06.995 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.994+0000 7f2db4ff9640 10 monclient(hunting): dump: 2024-03-29T11:33:06.995 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:06.995 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:06.995 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:06.995 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:06.995 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:06.995 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:06.995 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:06.995 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:06.995 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:06.995 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:06.995 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.994+0000 7f2db4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:06.995 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.995+0000 7f2db4ff9640 1 -- 172.21.15.135:0/2981836170 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3651960850 0 0) 0x7f2db81dbf70 con 0x7f2db81470b0 2024-03-29T11:33:06.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.995+0000 7f2db4ff9640 1 -- 172.21.15.135:0/2981836170 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d98004eb0 con 0x7f2db81470b0 2024-03-29T11:33:06.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.995+0000 7f2db4ff9640 1 -- 172.21.15.135:0/2981836170 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3841365454 0 0) 0x7f2db81219c0 con 0x7f2db80b4330 2024-03-29T11:33:06.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.995+0000 7f2db4ff9640 1 -- 172.21.15.135:0/2981836170 >> v1:172.21.15.194:6789/0 conn(0x7f2db81439b0 legacy=0x7f2db8140510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:06.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.995+0000 7f2db4ff9640 1 -- 172.21.15.135:0/2981836170 >> v1:172.21.15.135:6789/0 conn(0x7f2db81470b0 legacy=0x7f2db811f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:06.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.995+0000 7f2db4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:33:06.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.995+0000 7f2db4ff9640 1 monclient: found mon.c 2024-03-29T11:33:06.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.995+0000 7f2db4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:06.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.995+0000 7f2db4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:33:06.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.995+0000 7f2db4ff9640 1 -- 172.21.15.135:0/2981836170 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db8140c20 con 0x7f2db80b4330 2024-03-29T11:33:06.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.995+0000 7f2db4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:33:06.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.995+0000 7f2db4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:33:06.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.995+0000 7f2db4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:06.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.995+0000 7f2db4ff9640 10 client.?.objecter ms_handle_connect 0x7f2db81439b0 2024-03-29T11:33:06.997 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.995+0000 7f2db4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:06.997 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.995+0000 7f2dbe7af640 5 monclient: authenticate success, global_id 4556 2024-03-29T11:33:06.997 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.995+0000 7f2dbe7af640 10 monclient: _renew_subs 2024-03-29T11:33:06.997 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.995+0000 7f2dbe7af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:33:06.997 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.995+0000 7f2dbe7af640 1 -- 172.21.15.135:0/2981836170 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db8140ed0 con 0x7f2db80b4330 2024-03-29T11:33:06.997 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.995+0000 7f2dbe7af640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:06.997 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.995+0000 7f2dbe7af640 10 monclient: _renew_subs 2024-03-29T11:33:06.997 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.996+0000 7f2dbe7af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:33:06.997 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.996+0000 7f2dbe7af640 1 -- 172.21.15.135:0/2981836170 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db81412e0 con 0x7f2db80b4330 2024-03-29T11:33:06.997 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.996+0000 7f2dbe7af640 1 librados: init done 2024-03-29T11:33:06.999 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.999+0000 7f2db4ff9640 1 -- 172.21.15.135:0/2981836170 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2db00015f0 con 0x7f2db80b4330 2024-03-29T11:33:06.999 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.999+0000 7f2db4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:06.999 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:06.999+0000 7f2db67fc640 4 set_mon_vals no callback set 2024-03-29T11:33:06.999 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.000+0000 7f2db4ff9640 1 -- 172.21.15.135:0/2981836170 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f2db0004fb0 con 0x7f2db80b4330 2024-03-29T11:33:07.000 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.000+0000 7f2db4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:07.000 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.000+0000 7f2db4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:07.000 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.000+0000 7f2db4ff9640 10 monclient: dump: 2024-03-29T11:33:07.000 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:07.000 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:07.000 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:07.000 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:07.000 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:07.000 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:07.000 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:07.000 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:07.000 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:07.000 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:07.000 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.000+0000 7f2db4ff9640 1 -- 172.21.15.135:0/2981836170 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f2db00051d0 con 0x7f2db80b4330 2024-03-29T11:33:07.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.001+0000 7f2dbe7af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:07.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.001+0000 7f2dbe7af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:07.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.001+0000 7f2dbe7af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:33:07.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.001+0000 7f2dbe7af640 1 -- 172.21.15.135:0/2981836170 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d9c005080 con 0x7f2db80b4330 2024-03-29T11:33:07.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.001+0000 7f2db4ff9640 1 -- 172.21.15.135:0/2981836170 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f2db0084f30 con 0x7f2db80b4330 2024-03-29T11:33:07.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.001+0000 7f2db4ff9640 10 client.4556.objecter ms_dispatch 0x7f2db8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:07.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.001+0000 7f2db4ff9640 3 client.4556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:07.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.001+0000 7f2db4ff9640 3 client.4556.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:07.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.001+0000 7f2db4ff9640 20 client.4556.objecter dump_active .. 0 homeless 2024-03-29T11:33:07.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.004+0000 7f2db77fe640 10 monclient: get_auth_request con 0x7f2d9806ddf0 auth_method 0 2024-03-29T11:33:07.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.004+0000 7f2db4ff9640 10 client.4556.objecter ms_handle_connect 0x7f2d9806ddf0 2024-03-29T11:33:07.008 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.012+0000 7f2db4ff9640 1 -- 172.21.15.135:0/2981836170 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2db0050890 con 0x7f2db80b4330 2024-03-29T11:33:07.008 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.012+0000 7f2db4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:07.008 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.012+0000 7f2db4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:07.328 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.333+0000 7f2dbe7af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:07.329 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.333+0000 7f2dbe7af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:07.329 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.333+0000 7f2dbe7af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:33:07.329 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.333+0000 7f2dbe7af640 1 -- 172.21.15.135:0/2981836170 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d9c005330 con 0x7f2db80b4330 2024-03-29T11:33:07.332 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.335+0000 7f2db4ff9640 1 -- 172.21.15.135:0/2981836170 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f2db00527c0 con 0x7f2db80b4330 2024-03-29T11:33:07.332 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.335+0000 7f2db4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:07.332 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.335+0000 7f2db4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:07.333 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:33:07.333 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:07.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.340+0000 7f2dbe7af640 10 librados: watch_flush enter 2024-03-29T11:33:07.337 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.340+0000 7f2dbe7af640 10 librados: watch_flush exit 2024-03-29T11:33:07.337 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.340+0000 7f2dbe7af640 20 client.4556.objecter shutdown clearing up homeless session... 2024-03-29T11:33:07.337 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.340+0000 7f2dbe7af640 10 client.4556.objecter successfully canceled tick 2024-03-29T11:33:07.337 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.341+0000 7f2dbe7af640 1 -- 172.21.15.135:0/2981836170 >> v1:172.21.15.135:6816/3469875285 conn(0x7f2d9806ddf0 legacy=0x7f2d980702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:07.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.341+0000 7f2dbe7af640 10 monclient: shutdown 2024-03-29T11:33:07.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.341+0000 7f2dbe7af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:07.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.341+0000 7f2dbe7af640 1 -- 172.21.15.135:0/2981836170 >> v1:172.21.15.135:6790/0 conn(0x7f2db80b4330 legacy=0x7f2db81db860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:07.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.341+0000 7f2dbe7af640 1 -- 172.21.15.135:0/2981836170 shutdown_connections 2024-03-29T11:33:07.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.341+0000 7f2dbe7af640 1 -- 172.21.15.135:0/2981836170 >> 172.21.15.135:0/2981836170 conn(0x7f2db8066a00 msgr2=0x7f2db80b4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:07.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.342+0000 7f2dbe7af640 1 -- 172.21.15.135:0/2981836170 shutdown_connections 2024-03-29T11:33:07.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.342+0000 7f2dbe7af640 1 -- 172.21.15.135:0/2981836170 wait complete. 2024-03-29T11:33:07.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:07.342+0000 7f2dbe7af640 1 librados: shutdown 2024-03-29T11:33:07.358 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:14.164 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:14.295 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.299+0000 7f50133c6640 10 monclient: get_monmap_and_config 2024-03-29T11:33:14.295 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.299+0000 7f50133c6640 10 monclient: build_initial_monmap 2024-03-29T11:33:14.295 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.299+0000 7f50133c6640 10 monclient: monmap: 2024-03-29T11:33:14.296 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:33:14.296 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:14.296 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:33:14.300465+0000 2024-03-29T11:33:14.296 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:33:14.300465+0000 2024-03-29T11:33:14.296 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:14.296 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:14.296 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:33:14.296 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:33:14.296 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:33:14.296 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:14.297 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.300+0000 7f50133c6640 1 Processor -- start 2024-03-29T11:33:14.297 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.301+0000 7f50133c6640 1 -- start start 2024-03-29T11:33:14.297 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.301+0000 7f50133c6640 10 monclient: init 2024-03-29T11:33:14.298 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.301+0000 7f50133c6640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:14.298 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.301+0000 7f50133c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:33:14.298 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.301+0000 7f50133c6640 10 monclient(hunting): picked mon.noname-c con 0x7f500c1422b0 addr v1:172.21.15.194:6789/0 2024-03-29T11:33:14.298 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.301+0000 7f50133c6640 10 monclient(hunting): picked mon.noname-a con 0x7f500c142690 addr v1:172.21.15.135:6789/0 2024-03-29T11:33:14.298 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.301+0000 7f50133c6640 10 monclient(hunting): picked mon.noname-b con 0x7f500c0b17d0 addr v1:172.21.15.135:6790/0 2024-03-29T11:33:14.298 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.301+0000 7f50133c6640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500c142c80 con 0x7f500c142690 2024-03-29T11:33:14.298 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.301+0000 7f50133c6640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500c0aae40 con 0x7f500c1422b0 2024-03-29T11:33:14.298 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.301+0000 7f50133c6640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500c13fbd0 con 0x7f500c0b17d0 2024-03-29T11:33:14.298 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.301+0000 7f50133c6640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:14.299 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.301+0000 7f50133c6640 10 monclient(hunting): authenticate will time out at 1301.669928s 2024-03-29T11:33:14.299 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.302+0000 7f5011963640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f500c0b17d0 0x7f500c0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:58240/0 (socket says 172.21.15.135:58240) 2024-03-29T11:33:14.299 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.302+0000 7f5011963640 1 -- 172.21.15.135:0/3807777251 learned_addr learned my addr 172.21.15.135:0/3807777251 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:33:14.299 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.302+0000 7f5003fff640 1 -- 172.21.15.135:0/3807777251 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242046387 0 0) 0x7f500c13fbd0 con 0x7f500c0b17d0 2024-03-29T11:33:14.299 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.302+0000 7f5003fff640 10 monclient(hunting): _negotiate 2024-03-29T11:33:14.299 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.302+0000 7f5003fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:14.299 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.302+0000 7f5003fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:14.299 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.302+0000 7f5003fff640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:33:14.299 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.302+0000 7f5003fff640 1 -- 172.21.15.135:0/3807777251 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ff8003520 con 0x7f500c0b17d0 2024-03-29T11:33:14.299 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.302+0000 7f5003fff640 1 -- 172.21.15.135:0/3807777251 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1357266405 0 0) 0x7f4ff8003520 con 0x7f500c0b17d0 2024-03-29T11:33:14.299 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.302+0000 7f5003fff640 1 -- 172.21.15.135:0/3807777251 >> v1:172.21.15.194:6789/0 conn(0x7f500c1422b0 legacy=0x7f500c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:14.299 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.302+0000 7f5003fff640 1 -- 172.21.15.135:0/3807777251 >> v1:172.21.15.135:6789/0 conn(0x7f500c142690 legacy=0x7f500c0b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:33:14.300 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.303+0000 7f5003fff640 10 monclient: _finish_hunting 0 2024-03-29T11:33:14.300 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.303+0000 7f5003fff640 1 monclient: found mon.noname-b 2024-03-29T11:33:14.300 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.303+0000 7f5003fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:14.300 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.303+0000 7f5003fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:33:14.301 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.303+0000 7f5003fff640 1 -- 172.21.15.135:0/3807777251 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f500c151940 con 0x7f500c0b17d0 2024-03-29T11:33:14.301 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.303+0000 7f5003fff640 10 monclient: _finish_auth 0 2024-03-29T11:33:14.301 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.303+0000 7f5003fff640 10 monclient: _check_auth_tickets 2024-03-29T11:33:14.301 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.303+0000 7f5003fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:14.301 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.303+0000 7f5003fff640 1 -- 172.21.15.135:0/3807777251 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f4ffc001690 con 0x7f500c0b17d0 2024-03-29T11:33:14.301 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.304+0000 7f5003fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:14.301 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.304+0000 7f5003fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:33:14.301 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.304+0000 7f5003fff640 10 monclient: dump: 2024-03-29T11:33:14.301 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:14.301 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:14.301 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:14.301 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:14.302 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:14.302 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:14.302 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:14.302 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:14.302 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:14.302 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:14.302 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.304+0000 7f5003fff640 1 -- 172.21.15.135:0/3807777251 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ffc0017f0 con 0x7f500c0b17d0 2024-03-29T11:33:14.302 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.304+0000 7f5003fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:14.302 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.304+0000 7f5003fff640 1 -- 172.21.15.135:0/3807777251 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f4ffc001c40 con 0x7f500c0b17d0 2024-03-29T11:33:14.302 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.304+0000 7f5003fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:14.302 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.304+0000 7f5003fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:14.302 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.304+0000 7f5003fff640 10 monclient: dump: 2024-03-29T11:33:14.303 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:14.303 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:14.303 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:14.303 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:14.303 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:14.303 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:14.303 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:14.303 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:14.303 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:14.303 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:14.303 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.305+0000 7f50133c6640 5 monclient: authenticate success, global_id 4559 2024-03-29T11:33:14.303 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.305+0000 7f50133c6640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:14.303 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.305+0000 7f50133c6640 4 set_mon_vals no callback set 2024-03-29T11:33:14.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.306+0000 7f50133c6640 10 monclient: shutdown 2024-03-29T11:33:14.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.306+0000 7f50133c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:14.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.306+0000 7f50133c6640 1 -- 172.21.15.135:0/3807777251 >> v1:172.21.15.135:6790/0 conn(0x7f500c0b17d0 legacy=0x7f500c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:14.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.306+0000 7f50133c6640 1 -- 172.21.15.135:0/3807777251 shutdown_connections 2024-03-29T11:33:14.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.306+0000 7f50133c6640 1 -- 172.21.15.135:0/3807777251 >> 172.21.15.135:0/3807777251 conn(0x7f500c066a00 msgr2=0x7f500c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:14.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.306+0000 7f50133c6640 1 -- 172.21.15.135:0/3807777251 shutdown_connections 2024-03-29T11:33:14.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.306+0000 7f50133c6640 1 -- 172.21.15.135:0/3807777251 wait complete. 2024-03-29T11:33:14.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.307+0000 7f50133c6640 10 monclient: build_initial_monmap 2024-03-29T11:33:14.306 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.307+0000 7f50133c6640 10 monclient: monmap: 2024-03-29T11:33:14.306 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:33:14.306 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:14.307 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:33:14.307 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:33:14.307 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:14.307 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:14.307 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:33:14.307 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:33:14.307 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:33:14.307 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:14.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.307+0000 7f50133c6640 1 librados: starting msgr at 2024-03-29T11:33:14.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.307+0000 7f50133c6640 1 librados: starting objecter 2024-03-29T11:33:14.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.307+0000 7f50133c6640 1 Processor -- start 2024-03-29T11:33:14.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.307+0000 7f50133c6640 1 -- start start 2024-03-29T11:33:14.307 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.307+0000 7f50133c6640 1 librados: setting wanted keys 2024-03-29T11:33:14.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.307+0000 7f50133c6640 1 librados: calling monclient init 2024-03-29T11:33:14.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.307+0000 7f50133c6640 10 monclient: init 2024-03-29T11:33:14.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.308+0000 7f50133c6640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:14.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.308+0000 7f50133c6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:33:14.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.308+0000 7f50133c6640 10 monclient(hunting): picked mon.noname-c con 0x7f500c0b17d0 addr v1:172.21.15.135:6790/0 2024-03-29T11:33:14.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.308+0000 7f50133c6640 10 monclient(hunting): picked mon.noname-b con 0x7f500c1422b0 addr v1:172.21.15.194:6789/0 2024-03-29T11:33:14.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.308+0000 7f50133c6640 10 monclient(hunting): picked mon.noname-a con 0x7f500c142690 addr v1:172.21.15.135:6789/0 2024-03-29T11:33:14.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.308+0000 7f50133c6640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500c0ac9a0 con 0x7f500c142690 2024-03-29T11:33:14.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.308+0000 7f50133c6640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500c0acc80 con 0x7f500c1422b0 2024-03-29T11:33:14.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.308+0000 7f50133c6640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f500c0abd30 con 0x7f500c0b17d0 2024-03-29T11:33:14.308 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.308+0000 7f50133c6640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:14.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.308+0000 7f50133c6640 10 monclient(hunting): authenticate will time out at 1301.676808s 2024-03-29T11:33:14.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.308+0000 7f5011162640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f500c0b17d0 0x7f500c1daab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:52688/0 (socket says 172.21.15.135:52688) 2024-03-29T11:33:14.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.308+0000 7f5011162640 1 -- 172.21.15.135:0/932085587 learned_addr learned my addr 172.21.15.135:0/932085587 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:33:14.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.309+0000 7f5001ffb640 10 client.?.objecter ms_handle_connect 0x7f500c0b17d0 2024-03-29T11:33:14.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.309+0000 7f5001ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:14.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.309+0000 7f5001ffb640 1 -- 172.21.15.135:0/932085587 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960695490 0 0) 0x7f500c0abd30 con 0x7f500c0b17d0 2024-03-29T11:33:14.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.309+0000 7f5001ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:33:14.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.309+0000 7f5001ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:14.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.309+0000 7f5001ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:14.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.309+0000 7f5001ffb640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:33:14.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.309+0000 7f5001ffb640 1 -- 172.21.15.135:0/932085587 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ff0004eb0 con 0x7f500c0b17d0 2024-03-29T11:33:14.309 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.309+0000 7f5001ffb640 1 -- 172.21.15.135:0/932085587 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1144873906 0 0) 0x7f4ff0004eb0 con 0x7f500c0b17d0 2024-03-29T11:33:14.310 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.309+0000 7f5001ffb640 1 -- 172.21.15.135:0/932085587 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f500c0abd30 con 0x7f500c0b17d0 2024-03-29T11:33:14.310 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.309+0000 7f5001ffb640 1 -- 172.21.15.135:0/932085587 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f5008001960 con 0x7f500c0b17d0 2024-03-29T11:33:14.310 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.309+0000 7f5001ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:14.310 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.309+0000 7f5001ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:14.310 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.309+0000 7f5001ffb640 10 monclient(hunting): dump: 2024-03-29T11:33:14.310 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:14.310 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:14.310 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:14.310 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:14.310 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:14.310 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:14.311 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:14.311 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:14.311 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:14.311 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:14.311 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.309+0000 7f5001ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:14.311 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.310+0000 7f5001ffb640 1 -- 172.21.15.135:0/932085587 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2125314366 0 0) 0x7f500c0abd30 con 0x7f500c0b17d0 2024-03-29T11:33:14.311 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.310+0000 7f5001ffb640 1 -- 172.21.15.135:0/932085587 >> v1:172.21.15.194:6789/0 conn(0x7f500c1422b0 legacy=0x7f500c1db1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:14.311 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.310+0000 7f5001ffb640 1 -- 172.21.15.135:0/932085587 >> v1:172.21.15.135:6789/0 conn(0x7f500c142690 legacy=0x7f500c1e1bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:14.314 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.311+0000 7f5001ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:33:14.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.311+0000 7f5001ffb640 1 monclient: found mon.c 2024-03-29T11:33:14.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.311+0000 7f5001ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:14.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.311+0000 7f5001ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:33:14.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.311+0000 7f5001ffb640 1 -- 172.21.15.135:0/932085587 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f500c0aba10 con 0x7f500c0b17d0 2024-03-29T11:33:14.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.312+0000 7f5001ffb640 10 monclient: _finish_auth 0 2024-03-29T11:33:14.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.312+0000 7f5001ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:33:14.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.312+0000 7f5001ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:14.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.312+0000 7f50133c6640 5 monclient: authenticate success, global_id 4562 2024-03-29T11:33:14.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.312+0000 7f50133c6640 10 monclient: _renew_subs 2024-03-29T11:33:14.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.312+0000 7f50133c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:33:14.315 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.312+0000 7f50133c6640 1 -- 172.21.15.135:0/932085587 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f500c132910 con 0x7f500c0b17d0 2024-03-29T11:33:14.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.312+0000 7f50133c6640 10 client.4562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:14.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.312+0000 7f50133c6640 10 monclient: _renew_subs 2024-03-29T11:33:14.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.312+0000 7f50133c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:33:14.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.312+0000 7f50133c6640 1 -- 172.21.15.135:0/932085587 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f500c132560 con 0x7f500c0b17d0 2024-03-29T11:33:14.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.312+0000 7f50133c6640 1 librados: init done 2024-03-29T11:33:14.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.313+0000 7f5001ffb640 1 -- 172.21.15.135:0/932085587 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5008001bc0 con 0x7f500c0b17d0 2024-03-29T11:33:14.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.313+0000 7f5001ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:14.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.313+0000 7f50037fe640 4 set_mon_vals no callback set 2024-03-29T11:33:14.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.313+0000 7f50133c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:14.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.314+0000 7f5001ffb640 1 -- 172.21.15.135:0/932085587 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f5008003a40 con 0x7f500c0b17d0 2024-03-29T11:33:14.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.314+0000 7f50133c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:14.316 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.314+0000 7f50133c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:33:14.317 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.314+0000 7f50133c6640 1 -- 172.21.15.135:0/932085587 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fec005080 con 0x7f500c0b17d0 2024-03-29T11:33:14.317 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.314+0000 7f5001ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:14.317 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.314+0000 7f5001ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:14.317 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.314+0000 7f5001ffb640 10 monclient: dump: 2024-03-29T11:33:14.317 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:14.317 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:14.317 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:14.317 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:14.317 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:14.317 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:14.317 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:14.317 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:14.317 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:14.318 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:14.318 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.314+0000 7f5001ffb640 1 -- 172.21.15.135:0/932085587 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f5008003c60 con 0x7f500c0b17d0 2024-03-29T11:33:14.318 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.315+0000 7f5010961640 10 monclient: get_auth_request con 0x7f4ff006dca0 auth_method 0 2024-03-29T11:33:14.318 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.315+0000 7f5001ffb640 1 -- 172.21.15.135:0/932085587 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f5008084b80 con 0x7f500c0b17d0 2024-03-29T11:33:14.318 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.315+0000 7f5001ffb640 10 client.4562.objecter ms_dispatch 0x7f500c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:14.318 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.315+0000 7f5001ffb640 3 client.4562.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:14.318 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.316+0000 7f5001ffb640 3 client.4562.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:14.318 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.316+0000 7f5001ffb640 20 client.4562.objecter dump_active .. 0 homeless 2024-03-29T11:33:14.318 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.316+0000 7f5001ffb640 10 client.4562.objecter ms_handle_connect 0x7f4ff006dca0 2024-03-29T11:33:14.319 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.323+0000 7f5001ffb640 1 -- 172.21.15.135:0/932085587 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5008050480 con 0x7f500c0b17d0 2024-03-29T11:33:14.319 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.323+0000 7f5001ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:14.319 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.323+0000 7f5001ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:14.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.637+0000 7f50133c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:14.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.637+0000 7f50133c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:14.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.637+0000 7f50133c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:33:14.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.637+0000 7f50133c6640 1 -- 172.21.15.135:0/932085587 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4fec005600 con 0x7f500c0b17d0 2024-03-29T11:33:14.634 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.639+0000 7f5001ffb640 1 -- 172.21.15.135:0/932085587 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f50080523b0 con 0x7f500c0b17d0 2024-03-29T11:33:14.634 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.639+0000 7f5001ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:14.635 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.639+0000 7f5001ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:14.635 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:33:14.635 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:14.639 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.643+0000 7f50133c6640 10 librados: watch_flush enter 2024-03-29T11:33:14.639 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.643+0000 7f50133c6640 10 librados: watch_flush exit 2024-03-29T11:33:14.639 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.643+0000 7f50133c6640 20 client.4562.objecter shutdown clearing up homeless session... 2024-03-29T11:33:14.639 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.643+0000 7f50133c6640 10 client.4562.objecter successfully canceled tick 2024-03-29T11:33:14.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.644+0000 7f50133c6640 1 -- 172.21.15.135:0/932085587 >> v1:172.21.15.135:6816/3469875285 conn(0x7f4ff006dca0 legacy=0x7f4ff0070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:14.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.644+0000 7f50133c6640 10 monclient: shutdown 2024-03-29T11:33:14.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.644+0000 7f50133c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:14.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.644+0000 7f50133c6640 1 -- 172.21.15.135:0/932085587 >> v1:172.21.15.135:6790/0 conn(0x7f500c0b17d0 legacy=0x7f500c1daab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:14.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.644+0000 7f50133c6640 1 -- 172.21.15.135:0/932085587 shutdown_connections 2024-03-29T11:33:14.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.644+0000 7f50133c6640 1 -- 172.21.15.135:0/932085587 >> 172.21.15.135:0/932085587 conn(0x7f500c066a00 msgr2=0x7f500c0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:14.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.644+0000 7f50133c6640 1 -- 172.21.15.135:0/932085587 shutdown_connections 2024-03-29T11:33:14.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.644+0000 7f50133c6640 1 -- 172.21.15.135:0/932085587 wait complete. 2024-03-29T11:33:14.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:14.644+0000 7f50133c6640 1 librados: shutdown 2024-03-29T11:33:14.656 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:21.462 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:21.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.599+0000 7f3c78c4a640 10 monclient: get_monmap_and_config 2024-03-29T11:33:21.596 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.599+0000 7f3c78c4a640 10 monclient: build_initial_monmap 2024-03-29T11:33:21.596 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.599+0000 7f3c78c4a640 10 monclient: monmap: 2024-03-29T11:33:21.596 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:33:21.596 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:21.596 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:33:21.600537+0000 2024-03-29T11:33:21.596 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:33:21.600537+0000 2024-03-29T11:33:21.596 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:21.596 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:21.596 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:33:21.597 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:33:21.597 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:33:21.597 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:21.597 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.600+0000 7f3c78c4a640 1 Processor -- start 2024-03-29T11:33:21.597 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.600+0000 7f3c78c4a640 1 -- start start 2024-03-29T11:33:21.597 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.600+0000 7f3c78c4a640 10 monclient: init 2024-03-29T11:33:21.597 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.601+0000 7f3c78c4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:21.598 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.601+0000 7f3c78c4a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:33:21.598 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.601+0000 7f3c78c4a640 10 monclient(hunting): picked mon.noname-b con 0x7f3c740b4450 addr v1:172.21.15.135:6790/0 2024-03-29T11:33:21.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.601+0000 7f3c78c4a640 10 monclient(hunting): picked mon.noname-c con 0x7f3c7406e0a0 addr v1:172.21.15.194:6789/0 2024-03-29T11:33:21.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.601+0000 7f3c78c4a640 10 monclient(hunting): picked mon.noname-a con 0x7f3c74147110 addr v1:172.21.15.135:6789/0 2024-03-29T11:33:21.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.601+0000 7f3c78c4a640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c7406f030 con 0x7f3c74147110 2024-03-29T11:33:21.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.601+0000 7f3c78c4a640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c7406f310 con 0x7f3c7406e0a0 2024-03-29T11:33:21.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.601+0000 7f3c78c4a640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c741464e0 con 0x7f3c740b4450 2024-03-29T11:33:21.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.601+0000 7f3c78c4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:21.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.601+0000 7f3c78c4a640 10 monclient(hunting): authenticate will time out at 1308.970001s 2024-03-29T11:33:21.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.602+0000 7f3c7259c640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f3c740b4450 0x7f3c74146a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:46470/0 (socket says 172.21.15.135:46470) 2024-03-29T11:33:21.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.602+0000 7f3c7259c640 1 -- 172.21.15.135:0/4244446824 learned_addr learned my addr 172.21.15.135:0/4244446824 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:33:21.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.602+0000 7f3c7159a640 1 -- 172.21.15.135:0/4244446824 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3133516104 0 0) 0x7f3c741464e0 con 0x7f3c740b4450 2024-03-29T11:33:21.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.602+0000 7f3c7159a640 10 monclient(hunting): _negotiate 2024-03-29T11:33:21.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.602+0000 7f3c7159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:21.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.602+0000 7f3c7159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:21.599 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.602+0000 7f3c7159a640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:33:21.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.602+0000 7f3c7159a640 1 -- 172.21.15.135:0/4244446824 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c60003520 con 0x7f3c740b4450 2024-03-29T11:33:21.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.602+0000 7f3c7159a640 1 -- 172.21.15.135:0/4244446824 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 486120365 0 0) 0x7f3c7406f310 con 0x7f3c7406e0a0 2024-03-29T11:33:21.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.602+0000 7f3c7159a640 10 monclient(hunting): _negotiate 2024-03-29T11:33:21.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.602+0000 7f3c7159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:21.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.602+0000 7f3c7159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:21.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.602+0000 7f3c7159a640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:33:21.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.602+0000 7f3c7159a640 1 -- 172.21.15.135:0/4244446824 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c741464e0 con 0x7f3c7406e0a0 2024-03-29T11:33:21.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.602+0000 7f3c7159a640 1 -- 172.21.15.135:0/4244446824 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1997377419 0 0) 0x7f3c60003520 con 0x7f3c740b4450 2024-03-29T11:33:21.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.602+0000 7f3c7159a640 1 -- 172.21.15.135:0/4244446824 >> v1:172.21.15.194:6789/0 conn(0x7f3c7406e0a0 legacy=0x7f3c741432d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:21.600 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.602+0000 7f3c7159a640 1 -- 172.21.15.135:0/4244446824 >> v1:172.21.15.135:6789/0 conn(0x7f3c74147110 legacy=0x7f3c74151d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:33:21.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.604+0000 7f3c7159a640 10 monclient: _finish_hunting 0 2024-03-29T11:33:21.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.604+0000 7f3c7159a640 1 monclient: found mon.noname-b 2024-03-29T11:33:21.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.604+0000 7f3c7159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:21.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.604+0000 7f3c7159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:33:21.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.604+0000 7f3c7159a640 1 -- 172.21.15.135:0/4244446824 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c740b2590 con 0x7f3c740b4450 2024-03-29T11:33:21.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.604+0000 7f3c7159a640 10 monclient: _finish_auth 0 2024-03-29T11:33:21.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.604+0000 7f3c7159a640 10 monclient: _check_auth_tickets 2024-03-29T11:33:21.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.604+0000 7f3c7159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:21.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.604+0000 7f3c78c4a640 5 monclient: authenticate success, global_id 4565 2024-03-29T11:33:21.602 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.604+0000 7f3c78c4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:21.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.604+0000 7f3c7159a640 1 -- 172.21.15.135:0/4244446824 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f3c5c001690 con 0x7f3c740b4450 2024-03-29T11:33:21.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.604+0000 7f3c7159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:21.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.604+0000 7f3c7159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:33:21.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.604+0000 7f3c7159a640 10 monclient: dump: 2024-03-29T11:33:21.603 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:21.603 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:21.603 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:21.603 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:21.603 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:21.603 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:21.603 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:21.603 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:21.603 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:21.603 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:21.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.604+0000 7f3c78c4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:21.603 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.604+0000 7f3c7159a640 1 -- 172.21.15.135:0/4244446824 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c5c001870 con 0x7f3c740b4450 2024-03-29T11:33:21.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.604+0000 7f3c7159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:21.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.604+0000 7f3c78c4a640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:21.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.604+0000 7f3c78c4a640 4 set_mon_vals no callback set 2024-03-29T11:33:21.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.605+0000 7f3c7159a640 1 -- 172.21.15.135:0/4244446824 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f3c5c001c80 con 0x7f3c740b4450 2024-03-29T11:33:21.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.605+0000 7f3c7159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:21.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.605+0000 7f3c7159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:21.604 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.605+0000 7f3c7159a640 10 monclient: dump: 2024-03-29T11:33:21.604 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:21.604 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:21.604 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:21.604 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:21.604 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:21.604 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:21.604 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:21.605 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:21.605 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:21.605 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:21.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.605+0000 7f3c78c4a640 10 monclient: shutdown 2024-03-29T11:33:21.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.605+0000 7f3c78c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:21.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.605+0000 7f3c78c4a640 1 -- 172.21.15.135:0/4244446824 >> v1:172.21.15.135:6790/0 conn(0x7f3c740b4450 legacy=0x7f3c74146a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:21.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.606+0000 7f3c78c4a640 1 -- 172.21.15.135:0/4244446824 shutdown_connections 2024-03-29T11:33:21.605 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.606+0000 7f3c78c4a640 1 -- 172.21.15.135:0/4244446824 >> 172.21.15.135:0/4244446824 conn(0x7f3c74066a00 msgr2=0x7f3c74066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:21.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.606+0000 7f3c78c4a640 1 -- 172.21.15.135:0/4244446824 shutdown_connections 2024-03-29T11:33:21.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.607+0000 7f3c78c4a640 1 -- 172.21.15.135:0/4244446824 wait complete. 2024-03-29T11:33:21.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.608+0000 7f3c78c4a640 10 monclient: build_initial_monmap 2024-03-29T11:33:21.607 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.608+0000 7f3c78c4a640 10 monclient: monmap: 2024-03-29T11:33:21.607 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:33:21.607 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:21.607 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:33:21.607 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:33:21.607 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:21.608 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:21.608 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:33:21.608 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:33:21.608 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:33:21.608 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:21.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.608+0000 7f3c78c4a640 1 librados: starting msgr at 2024-03-29T11:33:21.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.608+0000 7f3c78c4a640 1 librados: starting objecter 2024-03-29T11:33:21.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.608+0000 7f3c78c4a640 1 Processor -- start 2024-03-29T11:33:21.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.608+0000 7f3c78c4a640 1 -- start start 2024-03-29T11:33:21.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.608+0000 7f3c78c4a640 1 librados: setting wanted keys 2024-03-29T11:33:21.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.608+0000 7f3c78c4a640 1 librados: calling monclient init 2024-03-29T11:33:21.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.608+0000 7f3c78c4a640 10 monclient: init 2024-03-29T11:33:21.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.609+0000 7f3c78c4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:21.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.609+0000 7f3c78c4a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:33:21.608 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.609+0000 7f3c78c4a640 10 monclient(hunting): picked mon.noname-b con 0x7f3c7406e0a0 addr v1:172.21.15.194:6789/0 2024-03-29T11:33:21.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.609+0000 7f3c78c4a640 10 monclient(hunting): picked mon.noname-a con 0x7f3c740b4450 addr v1:172.21.15.135:6789/0 2024-03-29T11:33:21.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.609+0000 7f3c78c4a640 10 monclient(hunting): picked mon.noname-c con 0x7f3c74147110 addr v1:172.21.15.135:6790/0 2024-03-29T11:33:21.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.609+0000 7f3c78c4a640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c74149b80 con 0x7f3c740b4450 2024-03-29T11:33:21.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.609+0000 7f3c78c4a640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c741dfa90 con 0x7f3c7406e0a0 2024-03-29T11:33:21.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.609+0000 7f3c78c4a640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c74149db0 con 0x7f3c74147110 2024-03-29T11:33:21.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.609+0000 7f3c78c4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:21.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.609+0000 7f3c78c4a640 10 monclient(hunting): authenticate will time out at 1308.977964s 2024-03-29T11:33:21.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.610+0000 7f3c72d9d640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f3c74147110 0x7f3c741e6ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:47246/0 (socket says 172.21.15.135:47246) 2024-03-29T11:33:21.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.610+0000 7f3c72d9d640 1 -- 172.21.15.135:0/4221489429 learned_addr learned my addr 172.21.15.135:0/4221489429 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:33:21.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.610+0000 7f3c6affd640 10 client.?.objecter ms_handle_connect 0x7f3c74147110 2024-03-29T11:33:21.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.610+0000 7f3c6affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:21.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.610+0000 7f3c6affd640 1 -- 172.21.15.135:0/4221489429 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305845033 0 0) 0x7f3c74149db0 con 0x7f3c74147110 2024-03-29T11:33:21.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.610+0000 7f3c6affd640 10 monclient(hunting): _negotiate 2024-03-29T11:33:21.609 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.610+0000 7f3c6affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:21.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.610+0000 7f3c6affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:21.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.610+0000 7f3c6affd640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:33:21.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.610+0000 7f3c6affd640 1 -- 172.21.15.135:0/4221489429 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c54004eb0 con 0x7f3c74147110 2024-03-29T11:33:21.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.610+0000 7f3c6affd640 1 -- 172.21.15.135:0/4221489429 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1667303552 0 0) 0x7f3c54004eb0 con 0x7f3c74147110 2024-03-29T11:33:21.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.610+0000 7f3c6affd640 1 -- 172.21.15.135:0/4221489429 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c74149db0 con 0x7f3c74147110 2024-03-29T11:33:21.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.610+0000 7f3c6affd640 1 -- 172.21.15.135:0/4221489429 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f3c6c001750 con 0x7f3c74147110 2024-03-29T11:33:21.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.610+0000 7f3c6affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:21.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.610+0000 7f3c6affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:21.610 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.610+0000 7f3c6affd640 10 monclient(hunting): dump: 2024-03-29T11:33:21.610 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:21.610 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:21.610 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:21.610 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:21.611 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:21.611 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:21.611 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:21.611 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:21.611 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:21.611 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:21.611 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.610+0000 7f3c6affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:21.611 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.611+0000 7f3c6affd640 1 -- 172.21.15.135:0/4221489429 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 813828701 0 0) 0x7f3c74149db0 con 0x7f3c74147110 2024-03-29T11:33:21.611 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.611+0000 7f3c6affd640 1 -- 172.21.15.135:0/4221489429 >> v1:172.21.15.194:6789/0 conn(0x7f3c7406e0a0 legacy=0x7f3c7414a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:21.611 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.611+0000 7f3c6affd640 1 -- 172.21.15.135:0/4221489429 >> v1:172.21.15.135:6789/0 conn(0x7f3c740b4450 legacy=0x7f3c741d8890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:33:21.613 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.612+0000 7f3c6affd640 10 monclient: _finish_hunting 0 2024-03-29T11:33:21.613 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.612+0000 7f3c6affd640 1 monclient: found mon.c 2024-03-29T11:33:21.613 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.612+0000 7f3c6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:21.613 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.612+0000 7f3c6affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:33:21.613 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.612+0000 7f3c6affd640 1 -- 172.21.15.135:0/4221489429 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c74149480 con 0x7f3c74147110 2024-03-29T11:33:21.613 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.612+0000 7f3c6affd640 10 monclient: _finish_auth 0 2024-03-29T11:33:21.613 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.612+0000 7f3c6affd640 10 monclient: _check_auth_tickets 2024-03-29T11:33:21.613 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.612+0000 7f3c6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:21.613 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.612+0000 7f3c78c4a640 5 monclient: authenticate success, global_id 4568 2024-03-29T11:33:21.613 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.612+0000 7f3c78c4a640 10 monclient: _renew_subs 2024-03-29T11:33:21.613 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.612+0000 7f3c78c4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:33:21.613 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.612+0000 7f3c78c4a640 1 -- 172.21.15.135:0/4221489429 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c74149760 con 0x7f3c74147110 2024-03-29T11:33:21.613 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.612+0000 7f3c78c4a640 10 client.4568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:21.614 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.612+0000 7f3c78c4a640 10 monclient: _renew_subs 2024-03-29T11:33:21.614 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.612+0000 7f3c78c4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:33:21.614 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.612+0000 7f3c78c4a640 1 -- 172.21.15.135:0/4221489429 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c741d91d0 con 0x7f3c74147110 2024-03-29T11:33:21.614 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.612+0000 7f3c78c4a640 1 librados: init done 2024-03-29T11:33:21.614 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.612+0000 7f3c6affd640 1 -- 172.21.15.135:0/4221489429 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c6c0018b0 con 0x7f3c74147110 2024-03-29T11:33:21.614 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.612+0000 7f3c6affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:21.614 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.612+0000 7f3c6affd640 1 -- 172.21.15.135:0/4221489429 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f3c6c0049d0 con 0x7f3c74147110 2024-03-29T11:33:21.614 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.612+0000 7f3c6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:21.614 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.612+0000 7f3c6affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:21.614 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.612+0000 7f3c6affd640 10 monclient: dump: 2024-03-29T11:33:21.614 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:21.614 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:21.614 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:21.614 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:21.615 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:21.615 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:21.615 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:21.615 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:21.615 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:21.615 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:21.615 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.612+0000 7f3c70d99640 4 set_mon_vals no callback set 2024-03-29T11:33:21.615 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.613+0000 7f3c78c4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:21.615 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.613+0000 7f3c78c4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:21.615 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.613+0000 7f3c78c4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:33:21.615 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.613+0000 7f3c78c4a640 1 -- 172.21.15.135:0/4221489429 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c58005080 con 0x7f3c74147110 2024-03-29T11:33:21.615 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.615+0000 7f3c6affd640 1 -- 172.21.15.135:0/4221489429 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f3c6c004b30 con 0x7f3c74147110 2024-03-29T11:33:21.615 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.615+0000 7f3c6affd640 1 -- 172.21.15.135:0/4221489429 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f3c6c084990 con 0x7f3c74147110 2024-03-29T11:33:21.615 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.615+0000 7f3c6affd640 10 client.4568.objecter ms_dispatch 0x7f3c74000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:21.615 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.615+0000 7f3c6affd640 3 client.4568.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:21.616 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.615+0000 7f3c6affd640 3 client.4568.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:21.616 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.615+0000 7f3c7259c640 10 monclient: get_auth_request con 0x7f3c5406dd00 auth_method 0 2024-03-29T11:33:21.616 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.615+0000 7f3c6affd640 20 client.4568.objecter dump_active .. 0 homeless 2024-03-29T11:33:21.616 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.616+0000 7f3c6affd640 10 client.4568.objecter ms_handle_connect 0x7f3c5406dd00 2024-03-29T11:33:21.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.622+0000 7f3c6affd640 1 -- 172.21.15.135:0/4221489429 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3c6c050270 con 0x7f3c74147110 2024-03-29T11:33:21.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.622+0000 7f3c6affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:21.618 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.622+0000 7f3c6affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:21.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.916+0000 7f3c78c4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:21.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.916+0000 7f3c78c4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:21.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.916+0000 7f3c78c4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:33:21.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.916+0000 7f3c78c4a640 1 -- 172.21.15.135:0/4221489429 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c58005330 con 0x7f3c74147110 2024-03-29T11:33:21.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.917+0000 7f3c6affd640 1 -- 172.21.15.135:0/4221489429 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f3c6c0521a0 con 0x7f3c74147110 2024-03-29T11:33:21.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.917+0000 7f3c6affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:21.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.917+0000 7f3c6affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:21.915 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:33:21.915 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:21.916 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.920+0000 7f3c78c4a640 10 librados: watch_flush enter 2024-03-29T11:33:21.917 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.920+0000 7f3c78c4a640 10 librados: watch_flush exit 2024-03-29T11:33:21.917 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.920+0000 7f3c78c4a640 20 client.4568.objecter shutdown clearing up homeless session... 2024-03-29T11:33:21.917 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.920+0000 7f3c78c4a640 10 client.4568.objecter successfully canceled tick 2024-03-29T11:33:21.917 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.920+0000 7f3c78c4a640 1 -- 172.21.15.135:0/4221489429 >> v1:172.21.15.135:6816/3469875285 conn(0x7f3c5406dd00 legacy=0x7f3c540701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:21.917 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.920+0000 7f3c78c4a640 10 monclient: shutdown 2024-03-29T11:33:21.917 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.920+0000 7f3c78c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:21.917 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.920+0000 7f3c78c4a640 1 -- 172.21.15.135:0/4221489429 >> v1:172.21.15.135:6790/0 conn(0x7f3c74147110 legacy=0x7f3c741e6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:21.917 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.920+0000 7f3c78c4a640 1 -- 172.21.15.135:0/4221489429 shutdown_connections 2024-03-29T11:33:21.917 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.920+0000 7f3c78c4a640 1 -- 172.21.15.135:0/4221489429 >> 172.21.15.135:0/4221489429 conn(0x7f3c74066a00 msgr2=0x7f3c74140e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:21.917 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.920+0000 7f3c78c4a640 1 -- 172.21.15.135:0/4221489429 shutdown_connections 2024-03-29T11:33:21.917 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.920+0000 7f3c78c4a640 1 -- 172.21.15.135:0/4221489429 wait complete. 2024-03-29T11:33:21.917 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:21.920+0000 7f3c78c4a640 1 librados: shutdown 2024-03-29T11:33:21.932 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:28.740 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:28.894 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.898+0000 7fb5f4b31640 10 monclient: get_monmap_and_config 2024-03-29T11:33:28.894 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.898+0000 7fb5f4b31640 10 monclient: build_initial_monmap 2024-03-29T11:33:28.894 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.898+0000 7fb5f4b31640 10 monclient: monmap: 2024-03-29T11:33:28.894 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:33:28.894 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:28.894 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:33:28.898841+0000 2024-03-29T11:33:28.894 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:33:28.898841+0000 2024-03-29T11:33:28.895 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:28.895 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:28.895 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:33:28.895 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:33:28.895 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:33:28.895 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:28.895 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.899+0000 7fb5f4b31640 1 Processor -- start 2024-03-29T11:33:28.896 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.899+0000 7fb5f4b31640 1 -- start start 2024-03-29T11:33:28.896 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.899+0000 7fb5f4b31640 10 monclient: init 2024-03-29T11:33:28.896 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.900+0000 7fb5f4b31640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:28.896 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.900+0000 7fb5f4b31640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:33:28.897 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.900+0000 7fb5f4b31640 10 monclient(hunting): picked mon.noname-c con 0x7fb5f00af390 addr v1:172.21.15.194:6789/0 2024-03-29T11:33:28.897 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.900+0000 7fb5f4b31640 10 monclient(hunting): picked mon.noname-a con 0x7fb5f00b3990 addr v1:172.21.15.135:6789/0 2024-03-29T11:33:28.898 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.900+0000 7fb5f4b31640 10 monclient(hunting): picked mon.noname-b con 0x7fb5f00b16d0 addr v1:172.21.15.135:6790/0 2024-03-29T11:33:28.898 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.900+0000 7fb5f4b31640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f00b47f0 con 0x7fb5f00b3990 2024-03-29T11:33:28.898 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.900+0000 7fb5f4b31640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f00b1b20 con 0x7fb5f00af390 2024-03-29T11:33:28.898 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.900+0000 7fb5f4b31640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f00b44a0 con 0x7fb5f00b16d0 2024-03-29T11:33:28.898 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.900+0000 7fb5f4b31640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:28.898 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.900+0000 7fb5f4b31640 10 monclient(hunting): authenticate will time out at 1316.268867s 2024-03-29T11:33:28.899 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.903+0000 7fb5ef7fe640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fb5f00af390 0x7fb5f01459f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:45323/0 (socket says 172.21.15.135:45323) 2024-03-29T11:33:28.899 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.903+0000 7fb5ef7fe640 1 -- 172.21.15.135:0/1993636409 learned_addr learned my addr 172.21.15.135:0/1993636409 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:33:28.902 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.904+0000 7fb5ee7fc640 1 -- 172.21.15.135:0/1993636409 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569686991 0 0) 0x7fb5f00b1b20 con 0x7fb5f00af390 2024-03-29T11:33:28.902 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.904+0000 7fb5ee7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:33:28.902 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.904+0000 7fb5ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:28.902 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.904+0000 7fb5ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:28.902 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.904+0000 7fb5ee7fc640 10 monclient(hunting): my global_id is 4543 2024-03-29T11:33:28.902 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.904+0000 7fb5ee7fc640 1 -- 172.21.15.135:0/1993636409 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5d8003520 con 0x7fb5f00af390 2024-03-29T11:33:28.902 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.906+0000 7fb5ee7fc640 1 -- 172.21.15.135:0/1993636409 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 792320549 0 0) 0x7fb5d8003520 con 0x7fb5f00af390 2024-03-29T11:33:28.902 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.906+0000 7fb5ee7fc640 1 -- 172.21.15.135:0/1993636409 >> v1:172.21.15.135:6790/0 conn(0x7fb5f00b16d0 legacy=0x7fb5f0151d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:28.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.906+0000 7fb5ee7fc640 1 -- 172.21.15.135:0/1993636409 >> v1:172.21.15.135:6789/0 conn(0x7fb5f00b3990 legacy=0x7fb5f00b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:28.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.906+0000 7fb5ee7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:33:28.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.906+0000 7fb5ee7fc640 1 monclient: found mon.noname-c 2024-03-29T11:33:28.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.906+0000 7fb5ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:28.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.906+0000 7fb5ee7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:33:28.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.906+0000 7fb5ee7fc640 1 -- 172.21.15.135:0/1993636409 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f00b4c80 con 0x7fb5f00af390 2024-03-29T11:33:28.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.906+0000 7fb5ee7fc640 10 monclient: _finish_auth 0 2024-03-29T11:33:28.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.906+0000 7fb5ee7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:33:28.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.906+0000 7fb5ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:28.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.906+0000 7fb5ee7fc640 1 -- 172.21.15.135:0/1993636409 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fb5e0001690 con 0x7fb5f00af390 2024-03-29T11:33:28.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.906+0000 7fb5ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:28.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.906+0000 7fb5ee7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:28.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.906+0000 7fb5ee7fc640 10 monclient: dump: 2024-03-29T11:33:28.903 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:28.904 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:28.904 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:28.904 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:28.904 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:28.904 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:28.904 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:28.904 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:28.904 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:28.904 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:28.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.906+0000 7fb5f4b31640 5 monclient: authenticate success, global_id 4543 2024-03-29T11:33:28.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.906+0000 7fb5f4b31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:28.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.906+0000 7fb5ee7fc640 1 -- 172.21.15.135:0/1993636409 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5e0001920 con 0x7fb5f00af390 2024-03-29T11:33:28.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.906+0000 7fb5ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:28.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.906+0000 7fb5ee7fc640 1 -- 172.21.15.135:0/1993636409 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fb5e0001690 con 0x7fb5f00af390 2024-03-29T11:33:28.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.906+0000 7fb5ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:28.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.906+0000 7fb5ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:33:28.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.906+0000 7fb5ee7fc640 10 monclient: dump: 2024-03-29T11:33:28.905 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:28.905 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:28.905 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:28.905 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:28.905 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:28.905 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:28.905 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:28.905 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:28.905 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:28.905 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:28.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.906+0000 7fb5f4b31640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:28.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.906+0000 7fb5f4b31640 4 set_mon_vals no callback set 2024-03-29T11:33:28.907 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.907+0000 7fb5f4b31640 10 monclient: shutdown 2024-03-29T11:33:28.907 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.907+0000 7fb5f4b31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:28.907 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.907+0000 7fb5f4b31640 1 -- 172.21.15.135:0/1993636409 >> v1:172.21.15.194:6789/0 conn(0x7fb5f00af390 legacy=0x7fb5f01459f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:28.907 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.907+0000 7fb5f4b31640 1 -- 172.21.15.135:0/1993636409 shutdown_connections 2024-03-29T11:33:28.907 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.907+0000 7fb5f4b31640 1 -- 172.21.15.135:0/1993636409 >> 172.21.15.135:0/1993636409 conn(0x7fb5f006d230 msgr2=0x7fb5f0066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:28.907 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.908+0000 7fb5f4b31640 1 -- 172.21.15.135:0/1993636409 shutdown_connections 2024-03-29T11:33:28.907 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.908+0000 7fb5f4b31640 1 -- 172.21.15.135:0/1993636409 wait complete. 2024-03-29T11:33:28.907 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.908+0000 7fb5f4b31640 10 monclient: build_initial_monmap 2024-03-29T11:33:28.907 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.908+0000 7fb5f4b31640 10 monclient: monmap: 2024-03-29T11:33:28.908 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:33:28.908 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:28.908 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:33:28.908 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:33:28.908 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:28.908 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:28.908 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:33:28.908 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:33:28.908 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:33:28.908 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:28.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.909+0000 7fb5f4b31640 1 librados: starting msgr at 2024-03-29T11:33:28.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.909+0000 7fb5f4b31640 1 librados: starting objecter 2024-03-29T11:33:28.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.909+0000 7fb5f4b31640 1 Processor -- start 2024-03-29T11:33:28.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.909+0000 7fb5f4b31640 1 -- start start 2024-03-29T11:33:28.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.910+0000 7fb5f4b31640 1 librados: setting wanted keys 2024-03-29T11:33:28.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.910+0000 7fb5f4b31640 1 librados: calling monclient init 2024-03-29T11:33:28.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.910+0000 7fb5f4b31640 10 monclient: init 2024-03-29T11:33:28.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.911+0000 7fb5f4b31640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:28.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.911+0000 7fb5f4b31640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:33:28.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.911+0000 7fb5f4b31640 10 monclient(hunting): picked mon.noname-a con 0x7fb5f00af390 addr v1:172.21.15.135:6789/0 2024-03-29T11:33:28.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.914+0000 7fb5ef7fe640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fb5f00af390 0x7fb5f0140570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:37663/0 (socket says 172.21.15.135:37663) 2024-03-29T11:33:28.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.914+0000 7fb5ef7fe640 1 -- 172.21.15.135:0/3925845381 learned_addr learned my addr 172.21.15.135:0/3925845381 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:33:28.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.915+0000 7fb5d7fff640 10 client.?.objecter ms_handle_connect 0x7fb5f00af390 2024-03-29T11:33:28.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.915+0000 7fb5d7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:28.912 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.912+0000 7fb5f4b31640 10 monclient(hunting): picked mon.noname-b con 0x7fb5f00b16d0 addr v1:172.21.15.194:6789/0 2024-03-29T11:33:28.912 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.915+0000 7fb5f4b31640 10 monclient(hunting): picked mon.noname-c con 0x7fb5f00b3990 addr v1:172.21.15.135:6790/0 2024-03-29T11:33:28.912 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.915+0000 7fb5f4b31640 1 -- 172.21.15.135:0/3925845381 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f013c1e0 con 0x7fb5f00af390 2024-03-29T11:33:28.912 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.915+0000 7fb5f4b31640 1 -- 172.21.15.135:0/3925845381 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f014a0a0 con 0x7fb5f00b16d0 2024-03-29T11:33:28.912 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.916+0000 7fb5f4b31640 1 -- 172.21.15.135:0/3925845381 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f013bf60 con 0x7fb5f00b3990 2024-03-29T11:33:28.912 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.916+0000 7fb5f4b31640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:28.912 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.916+0000 7fb5f4b31640 10 monclient(hunting): authenticate will time out at 1316.284099s 2024-03-29T11:33:28.912 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.916+0000 7fb5d7fff640 10 client.?.objecter ms_handle_connect 0x7fb5f00b3990 2024-03-29T11:33:28.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.916+0000 7fb5d7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:28.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.916+0000 7fb5d7fff640 1 -- 172.21.15.135:0/3925845381 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656610901 0 0) 0x7fb5f00b4c80 con 0x7fb5f00af390 2024-03-29T11:33:28.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.916+0000 7fb5d7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:33:28.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.916+0000 7fb5d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:28.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.916+0000 7fb5d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:28.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.916+0000 7fb5d7fff640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:33:28.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.916+0000 7fb5d7fff640 1 -- 172.21.15.135:0/3925845381 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5dc004eb0 con 0x7fb5f00af390 2024-03-29T11:33:28.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.918+0000 7fb5d7fff640 1 -- 172.21.15.135:0/3925845381 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1174567687 0 0) 0x7fb5f013bf60 con 0x7fb5f00b3990 2024-03-29T11:33:28.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.918+0000 7fb5d7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:33:28.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.918+0000 7fb5d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:28.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.918+0000 7fb5d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:28.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.918+0000 7fb5d7fff640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:33:28.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.918+0000 7fb5d7fff640 1 -- 172.21.15.135:0/3925845381 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5f00b4c80 con 0x7fb5f00b3990 2024-03-29T11:33:28.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.919+0000 7fb5d7fff640 10 client.?.objecter ms_handle_connect 0x7fb5f00b16d0 2024-03-29T11:33:28.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.919+0000 7fb5d7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:28.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.919+0000 7fb5d7fff640 1 -- 172.21.15.135:0/3925845381 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3645657075 0 0) 0x7fb5f00b4c80 con 0x7fb5f00b3990 2024-03-29T11:33:28.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.919+0000 7fb5d7fff640 1 -- 172.21.15.135:0/3925845381 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5f013bf60 con 0x7fb5f00b3990 2024-03-29T11:33:28.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.919+0000 7fb5d7fff640 1 -- 172.21.15.135:0/3925845381 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fb5e8001bf0 con 0x7fb5f00b3990 2024-03-29T11:33:28.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.919+0000 7fb5d7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:28.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.919+0000 7fb5d7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:28.915 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.919+0000 7fb5d7fff640 10 monclient(hunting): dump: 2024-03-29T11:33:28.915 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:28.916 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:28.916 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:28.916 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:28.916 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:28.916 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:28.916 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:28.916 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:28.916 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:28.916 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:28.916 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.919+0000 7fb5d7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:28.916 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.919+0000 7fb5d7fff640 1 -- 172.21.15.135:0/3925845381 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2590228625 0 0) 0x7fb5dc004eb0 con 0x7fb5f00af390 2024-03-29T11:33:28.916 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.919+0000 7fb5d7fff640 1 -- 172.21.15.135:0/3925845381 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5f00b4c80 con 0x7fb5f00af390 2024-03-29T11:33:28.916 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.919+0000 7fb5d7fff640 1 -- 172.21.15.135:0/3925845381 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054139128 0 0) 0x7fb5f014a0a0 con 0x7fb5f00b16d0 2024-03-29T11:33:28.916 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.919+0000 7fb5d7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:33:28.917 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.919+0000 7fb5d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:28.917 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.919+0000 7fb5d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:28.917 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.919+0000 7fb5d7fff640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:33:28.917 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.919+0000 7fb5d7fff640 1 -- 172.21.15.135:0/3925845381 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5dc004eb0 con 0x7fb5f00b16d0 2024-03-29T11:33:28.917 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.919+0000 7fb5d7fff640 1 -- 172.21.15.135:0/3925845381 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fb5e0001690 con 0x7fb5f00af390 2024-03-29T11:33:28.917 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.919+0000 7fb5d7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:28.917 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.919+0000 7fb5d7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:33:28.917 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.919+0000 7fb5d7fff640 10 monclient(hunting): dump: 2024-03-29T11:33:28.917 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:28.917 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:28.917 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:28.917 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:28.917 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:28.917 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:28.917 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:28.918 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:28.918 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:28.918 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:28.919 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.919+0000 7fb5d7fff640 1 -- 172.21.15.135:0/3925845381 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1058352998 0 0) 0x7fb5f013bf60 con 0x7fb5f00b3990 2024-03-29T11:33:28.919 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.920+0000 7fb5d7fff640 1 -- 172.21.15.135:0/3925845381 >> v1:172.21.15.194:6789/0 conn(0x7fb5f00b16d0 legacy=0x7fb5f013b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:28.919 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.920+0000 7fb5d7fff640 1 -- 172.21.15.135:0/3925845381 >> v1:172.21.15.135:6789/0 conn(0x7fb5f00af390 legacy=0x7fb5f0140570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:28.919 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.920+0000 7fb5d7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:33:28.919 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.920+0000 7fb5d7fff640 1 monclient: found mon.c 2024-03-29T11:33:28.919 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.920+0000 7fb5d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:28.919 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.920+0000 7fb5d7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:33:28.919 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.920+0000 7fb5d7fff640 1 -- 172.21.15.135:0/3925845381 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f014a380 con 0x7fb5f00b3990 2024-03-29T11:33:28.920 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.920+0000 7fb5d7fff640 10 monclient: _finish_auth 0 2024-03-29T11:33:28.920 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.920+0000 7fb5d7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:33:28.920 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.920+0000 7fb5d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:28.920 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.920+0000 7fb5d7fff640 1 -- 172.21.15.135:0/3925845381 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5e8003990 con 0x7fb5f00b3990 2024-03-29T11:33:28.920 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.920+0000 7fb5d7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:28.920 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.920+0000 7fb5d7fff640 1 -- 172.21.15.135:0/3925845381 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fb5e8003d20 con 0x7fb5f00b3990 2024-03-29T11:33:28.920 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.920+0000 7fb5d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:28.920 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.920+0000 7fb5d7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:28.920 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.920+0000 7fb5d7fff640 10 monclient: dump: 2024-03-29T11:33:28.920 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:28.920 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:28.920 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:28.920 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:28.920 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:28.921 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:28.921 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:28.921 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:28.921 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:28.921 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:28.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.920+0000 7fb5f4b31640 5 monclient: authenticate success, global_id 4574 2024-03-29T11:33:28.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.920+0000 7fb5f4b31640 10 monclient: _renew_subs 2024-03-29T11:33:28.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.920+0000 7fb5f4b31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:33:28.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.920+0000 7fb5f4b31640 1 -- 172.21.15.135:0/3925845381 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5f014a660 con 0x7fb5f00b3990 2024-03-29T11:33:28.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.921+0000 7fb5ed7fa640 4 set_mon_vals no callback set 2024-03-29T11:33:28.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.922+0000 7fb5f4b31640 10 client.4574.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:28.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.923+0000 7fb5f4b31640 10 monclient: _renew_subs 2024-03-29T11:33:28.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.923+0000 7fb5f4b31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:33:28.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.923+0000 7fb5f4b31640 1 -- 172.21.15.135:0/3925845381 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f0149730 con 0x7fb5f00b3990 2024-03-29T11:33:28.921 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.923+0000 7fb5f4b31640 1 librados: init done 2024-03-29T11:33:28.923 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.923+0000 7fb5d7fff640 1 -- 172.21.15.135:0/3925845381 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fb5e8003e80 con 0x7fb5f00b3990 2024-03-29T11:33:28.923 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.924+0000 7fb5d7fff640 1 -- 172.21.15.135:0/3925845381 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7fb5e8084c20 con 0x7fb5f00b3990 2024-03-29T11:33:28.923 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.924+0000 7fb5d7fff640 10 client.4574.objecter ms_dispatch 0x7fb5f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:28.923 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.924+0000 7fb5d7fff640 3 client.4574.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:28.923 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.924+0000 7fb5d7fff640 3 client.4574.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:28.923 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.924+0000 7fb5d7fff640 20 client.4574.objecter dump_active .. 0 homeless 2024-03-29T11:33:28.924 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.924+0000 7fb5f4b31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:28.924 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.924+0000 7fb5f4b31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:28.924 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.924+0000 7fb5f4b31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:33:28.924 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.924+0000 7fb5f4b31640 1 -- 172.21.15.135:0/3925845381 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5f004f730 con 0x7fb5f00b3990 2024-03-29T11:33:28.924 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.924+0000 7fb5ef7fe640 10 monclient: get_auth_request con 0x7fb5dc06dfa0 auth_method 0 2024-03-29T11:33:28.924 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.925+0000 7fb5d7fff640 10 client.4574.objecter ms_handle_connect 0x7fb5dc06dfa0 2024-03-29T11:33:28.928 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.932+0000 7fb5d7fff640 1 -- 172.21.15.135:0/3925845381 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb5e804f4d0 con 0x7fb5f00b3990 2024-03-29T11:33:28.928 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.932+0000 7fb5d7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:28.928 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:28.932+0000 7fb5d7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:29.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:29.262+0000 7fb5f4b31640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:29.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:29.262+0000 7fb5f4b31640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:29.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:29.262+0000 7fb5f4b31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:33:29.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:29.262+0000 7fb5f4b31640 1 -- 172.21.15.135:0/3925845381 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5f0074e10 con 0x7fb5f00b3990 2024-03-29T11:33:29.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:29.263+0000 7fb5d7fff640 1 -- 172.21.15.135:0/3925845381 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7fb5e8051400 con 0x7fb5f00b3990 2024-03-29T11:33:29.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:29.263+0000 7fb5d7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:29.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:29.263+0000 7fb5d7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:29.262 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:33:29.262 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:29.267 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:29.270+0000 7fb5f4b31640 10 librados: watch_flush enter 2024-03-29T11:33:29.268 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:29.271+0000 7fb5f4b31640 10 librados: watch_flush exit 2024-03-29T11:33:29.268 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:29.271+0000 7fb5f4b31640 20 client.4574.objecter shutdown clearing up homeless session... 2024-03-29T11:33:29.268 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:29.271+0000 7fb5f4b31640 10 client.4574.objecter successfully canceled tick 2024-03-29T11:33:29.268 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:29.272+0000 7fb5f4b31640 1 -- 172.21.15.135:0/3925845381 >> v1:172.21.15.135:6816/3469875285 conn(0x7fb5dc06dfa0 legacy=0x7fb5dc070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:29.268 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:29.272+0000 7fb5f4b31640 10 monclient: shutdown 2024-03-29T11:33:29.268 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:29.272+0000 7fb5f4b31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:29.268 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:29.272+0000 7fb5f4b31640 1 -- 172.21.15.135:0/3925845381 >> v1:172.21.15.135:6790/0 conn(0x7fb5f00b3990 legacy=0x7fb5f013b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:29.268 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:29.272+0000 7fb5f4b31640 1 -- 172.21.15.135:0/3925845381 shutdown_connections 2024-03-29T11:33:29.268 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:29.272+0000 7fb5f4b31640 1 -- 172.21.15.135:0/3925845381 >> 172.21.15.135:0/3925845381 conn(0x7fb5f006d230 msgr2=0x7fb5f00b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:29.268 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:29.272+0000 7fb5f4b31640 1 -- 172.21.15.135:0/3925845381 shutdown_connections 2024-03-29T11:33:29.269 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:29.273+0000 7fb5f4b31640 1 -- 172.21.15.135:0/3925845381 wait complete. 2024-03-29T11:33:29.269 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:29.273+0000 7fb5f4b31640 1 librados: shutdown 2024-03-29T11:33:29.293 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:36.100 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:36.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.254+0000 7f82d555e640 10 monclient: get_monmap_and_config 2024-03-29T11:33:36.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.254+0000 7f82d555e640 10 monclient: build_initial_monmap 2024-03-29T11:33:36.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.254+0000 7f82d555e640 10 monclient: monmap: 2024-03-29T11:33:36.251 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:33:36.251 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:36.252 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:33:36.255755+0000 2024-03-29T11:33:36.252 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:33:36.255755+0000 2024-03-29T11:33:36.252 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:36.252 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:36.252 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:33:36.252 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:33:36.252 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:33:36.252 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:36.252 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.255+0000 7f82d555e640 1 Processor -- start 2024-03-29T11:33:36.256 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.256+0000 7f82d555e640 1 -- start start 2024-03-29T11:33:36.256 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.256+0000 7f82d555e640 10 monclient: init 2024-03-29T11:33:36.256 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.256+0000 7f82d555e640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:36.256 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.256+0000 7f82d555e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:33:36.256 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.256+0000 7f82d555e640 10 monclient(hunting): picked mon.noname-c con 0x7f82d00b1ec0 addr v1:172.21.15.194:6789/0 2024-03-29T11:33:36.257 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.259+0000 7f82d555e640 10 monclient(hunting): picked mon.noname-a con 0x7f82d0143c40 addr v1:172.21.15.135:6789/0 2024-03-29T11:33:36.257 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.259+0000 7f82d555e640 10 monclient(hunting): picked mon.noname-b con 0x7f82d00b0420 addr v1:172.21.15.135:6790/0 2024-03-29T11:33:36.257 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.259+0000 7f82ce7fc640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f82d0143c40 0x7f82d0149780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:52473/0 (socket says 172.21.15.135:52473) 2024-03-29T11:33:36.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.259+0000 7f82ce7fc640 1 -- 172.21.15.135:0/837903674 learned_addr learned my addr 172.21.15.135:0/837903674 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:33:36.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.259+0000 7f82d555e640 1 -- 172.21.15.135:0/837903674 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d00b0940 con 0x7f82d0143c40 2024-03-29T11:33:36.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.260+0000 7f82d555e640 1 -- 172.21.15.135:0/837903674 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d00afb10 con 0x7f82d00b1ec0 2024-03-29T11:33:36.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.260+0000 7f82d555e640 1 -- 172.21.15.135:0/837903674 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d00afca0 con 0x7f82d00b0420 2024-03-29T11:33:36.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.260+0000 7f82d555e640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:36.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.260+0000 7f82d555e640 10 monclient(hunting): authenticate will time out at 1323.629449s 2024-03-29T11:33:36.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.261+0000 7f82cdffb640 1 -- 172.21.15.135:0/837903674 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1651678218 0 0) 0x7f82d00b0940 con 0x7f82d0143c40 2024-03-29T11:33:36.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.261+0000 7f82cdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:33:36.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.261+0000 7f82cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:36.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.261+0000 7f82cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:36.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.261+0000 7f82cdffb640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:33:36.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.261+0000 7f82cdffb640 1 -- 172.21.15.135:0/837903674 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b8003520 con 0x7f82d0143c40 2024-03-29T11:33:36.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.261+0000 7f82cdffb640 1 -- 172.21.15.135:0/837903674 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 314913125 0 0) 0x7f82d00afb10 con 0x7f82d00b1ec0 2024-03-29T11:33:36.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.261+0000 7f82cdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:33:36.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.261+0000 7f82cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:36.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.261+0000 7f82cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:36.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.261+0000 7f82cdffb640 10 monclient(hunting): my global_id is 4549 2024-03-29T11:33:36.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.261+0000 7f82cdffb640 1 -- 172.21.15.135:0/837903674 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82d00b0940 con 0x7f82d00b1ec0 2024-03-29T11:33:36.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.263+0000 7f82cdffb640 1 -- 172.21.15.135:0/837903674 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 776896806 0 0) 0x7f82d00b0940 con 0x7f82d00b1ec0 2024-03-29T11:33:36.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.263+0000 7f82cdffb640 1 -- 172.21.15.135:0/837903674 >> v1:172.21.15.135:6790/0 conn(0x7f82d00b0420 legacy=0x7f82d0151b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:33:36.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.263+0000 7f82cdffb640 1 -- 172.21.15.135:0/837903674 >> v1:172.21.15.135:6789/0 conn(0x7f82d0143c40 legacy=0x7f82d0149780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:36.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.263+0000 7f82cdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:33:36.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.263+0000 7f82cdffb640 1 monclient: found mon.noname-c 2024-03-29T11:33:36.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.263+0000 7f82cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:36.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.263+0000 7f82cdffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:33:36.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.263+0000 7f82cdffb640 1 -- 172.21.15.135:0/837903674 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d00b0fb0 con 0x7f82d00b1ec0 2024-03-29T11:33:36.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.263+0000 7f82cdffb640 10 monclient: _finish_auth 0 2024-03-29T11:33:36.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.263+0000 7f82cdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:33:36.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.263+0000 7f82cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:36.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.263+0000 7f82cdffb640 1 -- 172.21.15.135:0/837903674 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f82c0001520 con 0x7f82d00b1ec0 2024-03-29T11:33:36.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.263+0000 7f82cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:36.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.263+0000 7f82cdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:36.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.263+0000 7f82cdffb640 10 monclient: dump: 2024-03-29T11:33:36.264 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:36.265 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:36.265 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:36.265 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:36.265 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:36.265 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:36.265 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:36.265 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:36.265 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:36.265 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:36.265 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.264+0000 7f82d555e640 5 monclient: authenticate success, global_id 4549 2024-03-29T11:33:36.265 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.264+0000 7f82d555e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:36.265 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.265+0000 7f82cdffb640 1 -- 172.21.15.135:0/837903674 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82c0001680 con 0x7f82d00b1ec0 2024-03-29T11:33:36.265 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.265+0000 7f82cdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:36.265 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.265+0000 7f82cdffb640 1 -- 172.21.15.135:0/837903674 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f82c0001ad0 con 0x7f82d00b1ec0 2024-03-29T11:33:36.266 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.265+0000 7f82cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:36.266 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.265+0000 7f82cdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:33:36.266 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.265+0000 7f82cdffb640 10 monclient: dump: 2024-03-29T11:33:36.266 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:36.266 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:36.266 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:36.266 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:36.266 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:36.266 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:36.266 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:36.266 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:36.266 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:36.266 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:36.266 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.265+0000 7f82d555e640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:36.267 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.265+0000 7f82d555e640 4 set_mon_vals no callback set 2024-03-29T11:33:36.267 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.266+0000 7f82d555e640 10 monclient: shutdown 2024-03-29T11:33:36.267 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.266+0000 7f82d555e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:36.267 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.266+0000 7f82d555e640 1 -- 172.21.15.135:0/837903674 >> v1:172.21.15.194:6789/0 conn(0x7f82d00b1ec0 legacy=0x7f82d0143530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:36.267 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.266+0000 7f82d555e640 1 -- 172.21.15.135:0/837903674 shutdown_connections 2024-03-29T11:33:36.267 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.266+0000 7f82d555e640 1 -- 172.21.15.135:0/837903674 >> 172.21.15.135:0/837903674 conn(0x7f82d006cb30 msgr2=0x7f82d006cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:36.267 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.270+0000 7f82d555e640 1 -- 172.21.15.135:0/837903674 shutdown_connections 2024-03-29T11:33:36.267 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.270+0000 7f82d555e640 1 -- 172.21.15.135:0/837903674 wait complete. 2024-03-29T11:33:36.269 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.271+0000 7f82d555e640 10 monclient: build_initial_monmap 2024-03-29T11:33:36.269 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.271+0000 7f82d555e640 10 monclient: monmap: 2024-03-29T11:33:36.269 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:33:36.269 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:36.269 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:33:36.270 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:33:36.270 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:36.270 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:36.270 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:33:36.270 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:33:36.270 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:33:36.270 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:36.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.272+0000 7f82d555e640 1 librados: starting msgr at 2024-03-29T11:33:36.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.272+0000 7f82d555e640 1 librados: starting objecter 2024-03-29T11:33:36.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.272+0000 7f82d555e640 1 Processor -- start 2024-03-29T11:33:36.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.272+0000 7f82d555e640 1 -- start start 2024-03-29T11:33:36.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.272+0000 7f82d555e640 1 librados: setting wanted keys 2024-03-29T11:33:36.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.272+0000 7f82d555e640 1 librados: calling monclient init 2024-03-29T11:33:36.270 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.272+0000 7f82d555e640 10 monclient: init 2024-03-29T11:33:36.271 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.273+0000 7f82d555e640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:36.271 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.273+0000 7f82d555e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:33:36.274 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.275+0000 7f82d555e640 10 monclient(hunting): picked mon.noname-c con 0x7f82d00b0420 addr v1:172.21.15.135:6790/0 2024-03-29T11:33:36.274 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.277+0000 7f82ceffd640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f82d00b0420 0x7f82d0138a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:43799/0 (socket says 172.21.15.135:43799) 2024-03-29T11:33:36.274 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.277+0000 7f82ceffd640 1 -- 172.21.15.135:0/3877425112 learned_addr learned my addr 172.21.15.135:0/3877425112 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:33:36.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.277+0000 7f82d555e640 10 monclient(hunting): picked mon.noname-b con 0x7f82d00b1ec0 addr v1:172.21.15.194:6789/0 2024-03-29T11:33:36.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.277+0000 7f82b77fe640 10 client.?.objecter ms_handle_connect 0x7f82d00b0420 2024-03-29T11:33:36.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.277+0000 7f82b77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:36.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.277+0000 7f82d555e640 10 monclient(hunting): picked mon.noname-a con 0x7f82d0143c40 addr v1:172.21.15.135:6789/0 2024-03-29T11:33:36.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.277+0000 7f82d555e640 1 -- 172.21.15.135:0/3877425112 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d0142110 con 0x7f82d0143c40 2024-03-29T11:33:36.275 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.277+0000 7f82d555e640 1 -- 172.21.15.135:0/3877425112 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d0148d00 con 0x7f82d00b1ec0 2024-03-29T11:33:36.277 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.280+0000 7f82d555e640 1 -- 172.21.15.135:0/3877425112 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d0142360 con 0x7f82d00b0420 2024-03-29T11:33:36.278 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.281+0000 7f82d555e640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:36.278 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.281+0000 7f82d555e640 10 monclient(hunting): authenticate will time out at 1323.650200s 2024-03-29T11:33:36.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.281+0000 7f82b77fe640 1 -- 172.21.15.135:0/3877425112 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2078748377 0 0) 0x7f82d0142360 con 0x7f82d00b0420 2024-03-29T11:33:36.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.281+0000 7f82b77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:33:36.282 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.281+0000 7f82b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:36.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.281+0000 7f82b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:36.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.281+0000 7f82b77fe640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:33:36.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.281+0000 7f82b77fe640 1 -- 172.21.15.135:0/3877425112 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82bc004eb0 con 0x7f82d00b0420 2024-03-29T11:33:36.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.281+0000 7f82b77fe640 10 client.?.objecter ms_handle_connect 0x7f82d00b1ec0 2024-03-29T11:33:36.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.281+0000 7f82b77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:36.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.282+0000 7f82b77fe640 1 -- 172.21.15.135:0/3877425112 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4057605156 0 0) 0x7f82bc004eb0 con 0x7f82d00b0420 2024-03-29T11:33:36.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.282+0000 7f82b77fe640 1 -- 172.21.15.135:0/3877425112 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82d0142360 con 0x7f82d00b0420 2024-03-29T11:33:36.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.282+0000 7f82b77fe640 1 -- 172.21.15.135:0/3877425112 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f82c0001a90 con 0x7f82d00b0420 2024-03-29T11:33:36.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.282+0000 7f82b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:36.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.282+0000 7f82b77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:36.283 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.282+0000 7f82b77fe640 10 monclient(hunting): dump: 2024-03-29T11:33:36.283 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:36.283 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:36.284 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:36.284 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:36.284 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:36.284 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:36.284 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:36.284 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:36.284 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:36.284 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:36.284 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.282+0000 7f82b77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:36.284 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.282+0000 7f82b77fe640 1 -- 172.21.15.135:0/3877425112 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1263287074 0 0) 0x7f82d0142360 con 0x7f82d00b0420 2024-03-29T11:33:36.284 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.282+0000 7f82b77fe640 1 -- 172.21.15.135:0/3877425112 >> v1:172.21.15.194:6789/0 conn(0x7f82d00b1ec0 legacy=0x7f82d01de6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:36.284 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.282+0000 7f82b77fe640 1 -- 172.21.15.135:0/3877425112 >> v1:172.21.15.135:6789/0 conn(0x7f82d0143c40 legacy=0x7f82d01e6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:36.285 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.282+0000 7f82b77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:33:36.285 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.282+0000 7f82b77fe640 1 monclient: found mon.c 2024-03-29T11:33:36.285 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.282+0000 7f82b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:36.285 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.282+0000 7f82b77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:33:36.285 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.282+0000 7f82b77fe640 1 -- 172.21.15.135:0/3877425112 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d0144880 con 0x7f82d00b0420 2024-03-29T11:33:36.285 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.282+0000 7f82b77fe640 10 monclient: _finish_auth 0 2024-03-29T11:33:36.285 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.282+0000 7f82b77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:33:36.285 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.282+0000 7f82b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:36.285 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.283+0000 7f82d555e640 5 monclient: authenticate success, global_id 4577 2024-03-29T11:33:36.285 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.283+0000 7f82d555e640 10 monclient: _renew_subs 2024-03-29T11:33:36.285 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.283+0000 7f82d555e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:33:36.285 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.283+0000 7f82d555e640 1 -- 172.21.15.135:0/3877425112 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d0144a70 con 0x7f82d00b0420 2024-03-29T11:33:36.286 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.283+0000 7f82d555e640 10 client.4577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:36.286 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.283+0000 7f82d555e640 10 monclient: _renew_subs 2024-03-29T11:33:36.286 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.283+0000 7f82d555e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:33:36.286 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.283+0000 7f82d555e640 1 -- 172.21.15.135:0/3877425112 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d0149670 con 0x7f82d00b0420 2024-03-29T11:33:36.286 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.283+0000 7f82d555e640 1 librados: init done 2024-03-29T11:33:36.286 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.284+0000 7f82b77fe640 1 -- 172.21.15.135:0/3877425112 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82c00010a0 con 0x7f82d00b0420 2024-03-29T11:33:36.286 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.284+0000 7f82b77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:36.286 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.284+0000 7f82b77fe640 1 -- 172.21.15.135:0/3877425112 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f82c0003b00 con 0x7f82d00b0420 2024-03-29T11:33:36.286 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.284+0000 7f82b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:36.286 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.284+0000 7f82b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:36.287 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.284+0000 7f82b77fe640 10 monclient: dump: 2024-03-29T11:33:36.287 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:36.287 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:36.287 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:36.287 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:36.287 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:36.287 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:36.287 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:36.287 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:36.287 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:36.288 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:36.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.284+0000 7f82d555e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:36.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.284+0000 7f82d555e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:36.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.284+0000 7f82d555e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:33:36.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.284+0000 7f82d555e640 1 -- 172.21.15.135:0/3877425112 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82d0075f10 con 0x7f82d00b0420 2024-03-29T11:33:36.288 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.284+0000 7f82ccff9640 4 set_mon_vals no callback set 2024-03-29T11:33:36.289 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.286+0000 7f82b77fe640 1 -- 172.21.15.135:0/3877425112 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f82c0003c60 con 0x7f82d00b0420 2024-03-29T11:33:36.290 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.287+0000 7f82b77fe640 1 -- 172.21.15.135:0/3877425112 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f82c0084ba0 con 0x7f82d00b0420 2024-03-29T11:33:36.290 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.287+0000 7f82b77fe640 10 client.4577.objecter ms_dispatch 0x7f82d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:36.290 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.287+0000 7f82b77fe640 3 client.4577.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:36.290 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.287+0000 7f82b77fe640 3 client.4577.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:36.290 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.287+0000 7f82b77fe640 20 client.4577.objecter dump_active .. 0 homeless 2024-03-29T11:33:36.290 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.291+0000 7f82ce7fc640 10 monclient: get_auth_request con 0x7f82bc06dd90 auth_method 0 2024-03-29T11:33:36.290 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.291+0000 7f82b77fe640 10 client.4577.objecter ms_handle_connect 0x7f82bc06dd90 2024-03-29T11:33:36.293 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.296+0000 7f82b77fe640 1 -- 172.21.15.135:0/3877425112 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f82c00502b0 con 0x7f82d00b0420 2024-03-29T11:33:36.293 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.296+0000 7f82b77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:36.294 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.296+0000 7f82b77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:36.624 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.627+0000 7f82d555e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:36.624 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.627+0000 7f82d555e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:36.624 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.627+0000 7f82d555e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:33:36.624 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.627+0000 7f82d555e640 1 -- 172.21.15.135:0/3877425112 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82d013f250 con 0x7f82d00b0420 2024-03-29T11:33:36.627 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.629+0000 7f82b77fe640 1 -- 172.21.15.135:0/3877425112 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f82c00523b0 con 0x7f82d00b0420 2024-03-29T11:33:36.627 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.630+0000 7f82b77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:36.627 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.630+0000 7f82b77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:36.627 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:33:36.627 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:36.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.634+0000 7f82d555e640 10 librados: watch_flush enter 2024-03-29T11:33:36.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.634+0000 7f82d555e640 10 librados: watch_flush exit 2024-03-29T11:33:36.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.634+0000 7f82d555e640 20 client.4577.objecter shutdown clearing up homeless session... 2024-03-29T11:33:36.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.634+0000 7f82d555e640 10 client.4577.objecter successfully canceled tick 2024-03-29T11:33:36.635 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.638+0000 7f82d555e640 1 -- 172.21.15.135:0/3877425112 >> v1:172.21.15.135:6816/3469875285 conn(0x7f82bc06dd90 legacy=0x7f82bc070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:36.635 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.638+0000 7f82d555e640 10 monclient: shutdown 2024-03-29T11:33:36.636 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.638+0000 7f82d555e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:36.636 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.638+0000 7f82d555e640 1 -- 172.21.15.135:0/3877425112 >> v1:172.21.15.135:6790/0 conn(0x7f82d00b0420 legacy=0x7f82d0138a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:36.636 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.638+0000 7f82d555e640 1 -- 172.21.15.135:0/3877425112 shutdown_connections 2024-03-29T11:33:36.636 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.638+0000 7f82d555e640 1 -- 172.21.15.135:0/3877425112 >> 172.21.15.135:0/3877425112 conn(0x7f82d006cb30 msgr2=0x7f82d00b2b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:36.636 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.638+0000 7f82d555e640 1 -- 172.21.15.135:0/3877425112 shutdown_connections 2024-03-29T11:33:36.636 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.638+0000 7f82d555e640 1 -- 172.21.15.135:0/3877425112 wait complete. 2024-03-29T11:33:36.636 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:36.638+0000 7f82d555e640 1 librados: shutdown 2024-03-29T11:33:36.656 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:43.463 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:43.614 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.616+0000 7f2dda84a640 10 monclient: get_monmap_and_config 2024-03-29T11:33:43.614 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.616+0000 7f2dda84a640 10 monclient: build_initial_monmap 2024-03-29T11:33:43.614 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.616+0000 7f2dda84a640 10 monclient: monmap: 2024-03-29T11:33:43.614 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:33:43.614 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:43.615 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:33:43.618488+0000 2024-03-29T11:33:43.615 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:33:43.618488+0000 2024-03-29T11:33:43.615 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:43.615 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:43.615 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:33:43.615 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:33:43.615 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:33:43.615 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:43.615 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.618+0000 7f2dda84a640 1 Processor -- start 2024-03-29T11:33:43.616 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.618+0000 7f2dda84a640 1 -- start start 2024-03-29T11:33:43.616 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.618+0000 7f2dda84a640 10 monclient: init 2024-03-29T11:33:43.616 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.619+0000 7f2dda84a640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:43.621 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.619+0000 7f2dda84a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:33:43.621 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.619+0000 7f2dda84a640 10 monclient(hunting): picked mon.noname-c con 0x7f2dd41457f0 addr v1:172.21.15.194:6789/0 2024-03-29T11:33:43.621 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.619+0000 7f2dda84a640 10 monclient(hunting): picked mon.noname-a con 0x7f2dd413f650 addr v1:172.21.15.135:6789/0 2024-03-29T11:33:43.621 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.619+0000 7f2dda84a640 10 monclient(hunting): picked mon.noname-b con 0x7f2dd40a9d70 addr v1:172.21.15.135:6790/0 2024-03-29T11:33:43.621 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.619+0000 7f2dda84a640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd40aaef0 con 0x7f2dd413f650 2024-03-29T11:33:43.621 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.619+0000 7f2dda84a640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd4145bd0 con 0x7f2dd41457f0 2024-03-29T11:33:43.622 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.619+0000 7f2dda84a640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd4145eb0 con 0x7f2dd40a9d70 2024-03-29T11:33:43.622 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.619+0000 7f2dda84a640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:43.622 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.619+0000 7f2dda84a640 10 monclient(hunting): authenticate will time out at 1330.988768s 2024-03-29T11:33:43.622 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.623+0000 7f2dd8de7640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f2dd40a9d70 0x7f2dd414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:59334/0 (socket says 172.21.15.135:59334) 2024-03-29T11:33:43.622 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.623+0000 7f2dd8de7640 1 -- 172.21.15.135:0/2547988696 learned_addr learned my addr 172.21.15.135:0/2547988696 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:33:43.624 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.626+0000 7f2dd2ffd640 1 -- 172.21.15.135:0/2547988696 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409768707 0 0) 0x7f2dd4145eb0 con 0x7f2dd40a9d70 2024-03-29T11:33:43.624 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.626+0000 7f2dd2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:33:43.624 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.626+0000 7f2dd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:43.624 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.626+0000 7f2dd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:43.624 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.626+0000 7f2dd2ffd640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:33:43.624 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.626+0000 7f2dd2ffd640 1 -- 172.21.15.135:0/2547988696 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dbc003520 con 0x7f2dd40a9d70 2024-03-29T11:33:43.629 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.628+0000 7f2dd2ffd640 1 -- 172.21.15.135:0/2547988696 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591110647 0 0) 0x7f2dd4145bd0 con 0x7f2dd41457f0 2024-03-29T11:33:43.629 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.628+0000 7f2dd2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:33:43.629 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.628+0000 7f2dd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:43.629 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.628+0000 7f2dd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:43.629 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.628+0000 7f2dd2ffd640 10 monclient(hunting): my global_id is 4555 2024-03-29T11:33:43.629 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.628+0000 7f2dd2ffd640 1 -- 172.21.15.135:0/2547988696 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dd4145eb0 con 0x7f2dd41457f0 2024-03-29T11:33:43.629 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.628+0000 7f2dd2ffd640 1 -- 172.21.15.135:0/2547988696 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1989684655 0 0) 0x7f2dbc003520 con 0x7f2dd40a9d70 2024-03-29T11:33:43.629 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.628+0000 7f2dd2ffd640 1 -- 172.21.15.135:0/2547988696 >> v1:172.21.15.194:6789/0 conn(0x7f2dd41457f0 legacy=0x7f2dd413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:43.629 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.628+0000 7f2dd2ffd640 1 -- 172.21.15.135:0/2547988696 >> v1:172.21.15.135:6789/0 conn(0x7f2dd413f650 legacy=0x7f2dd40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:43.629 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.628+0000 7f2dd2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:33:43.630 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.628+0000 7f2dd2ffd640 1 monclient: found mon.noname-b 2024-03-29T11:33:43.630 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.628+0000 7f2dd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:43.630 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.628+0000 7f2dd2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:33:43.630 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.628+0000 7f2dd2ffd640 1 -- 172.21.15.135:0/2547988696 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd4146340 con 0x7f2dd40a9d70 2024-03-29T11:33:43.630 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.629+0000 7f2dd2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:33:43.630 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.629+0000 7f2dd2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:33:43.630 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.629+0000 7f2dd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:43.630 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.629+0000 7f2dd2ffd640 1 -- 172.21.15.135:0/2547988696 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f2dcc001a10 con 0x7f2dd40a9d70 2024-03-29T11:33:43.630 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.629+0000 7f2dd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:43.630 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.629+0000 7f2dd2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:33:43.630 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.629+0000 7f2dd2ffd640 10 monclient: dump: 2024-03-29T11:33:43.630 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:43.630 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:43.630 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:43.631 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:43.631 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:43.631 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:43.631 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:43.631 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:43.631 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:43.631 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:43.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.629+0000 7f2dda84a640 5 monclient: authenticate success, global_id 4580 2024-03-29T11:33:43.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.629+0000 7f2dda84a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:43.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.629+0000 7f2dd2ffd640 1 -- 172.21.15.135:0/2547988696 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2dcc001c30 con 0x7f2dd40a9d70 2024-03-29T11:33:43.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.629+0000 7f2dd2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:43.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.629+0000 7f2dd2ffd640 1 -- 172.21.15.135:0/2547988696 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f2dcc001fc0 con 0x7f2dd40a9d70 2024-03-29T11:33:43.631 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.629+0000 7f2dda84a640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:43.632 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.629+0000 7f2dda84a640 4 set_mon_vals no callback set 2024-03-29T11:33:43.632 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.630+0000 7f2dd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:43.632 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.630+0000 7f2dd2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:43.632 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.630+0000 7f2dd2ffd640 10 monclient: dump: 2024-03-29T11:33:43.632 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:43.632 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:43.632 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:43.632 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:43.632 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:43.632 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:43.632 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:43.632 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:43.632 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:43.632 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:43.632 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.630+0000 7f2dda84a640 10 monclient: shutdown 2024-03-29T11:33:43.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.630+0000 7f2dda84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:43.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.630+0000 7f2dda84a640 1 -- 172.21.15.135:0/2547988696 >> v1:172.21.15.135:6790/0 conn(0x7f2dd40a9d70 legacy=0x7f2dd414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:43.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.631+0000 7f2dda84a640 1 -- 172.21.15.135:0/2547988696 shutdown_connections 2024-03-29T11:33:43.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.631+0000 7f2dda84a640 1 -- 172.21.15.135:0/2547988696 >> 172.21.15.135:0/2547988696 conn(0x7f2dd4066a00 msgr2=0x7f2dd4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:43.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.631+0000 7f2dda84a640 1 -- 172.21.15.135:0/2547988696 shutdown_connections 2024-03-29T11:33:43.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.631+0000 7f2dda84a640 1 -- 172.21.15.135:0/2547988696 wait complete. 2024-03-29T11:33:43.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.632+0000 7f2dda84a640 10 monclient: build_initial_monmap 2024-03-29T11:33:43.633 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.632+0000 7f2dda84a640 10 monclient: monmap: 2024-03-29T11:33:43.633 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:33:43.633 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:43.633 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:33:43.633 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:33:43.633 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:43.633 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:43.634 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:33:43.634 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:33:43.634 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:33:43.634 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:43.635 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.632+0000 7f2dda84a640 1 librados: starting msgr at 2024-03-29T11:33:43.635 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.632+0000 7f2dda84a640 1 librados: starting objecter 2024-03-29T11:33:43.635 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.633+0000 7f2dda84a640 1 Processor -- start 2024-03-29T11:33:43.635 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.633+0000 7f2dda84a640 1 -- start start 2024-03-29T11:33:43.635 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.633+0000 7f2dda84a640 1 librados: setting wanted keys 2024-03-29T11:33:43.635 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.633+0000 7f2dda84a640 1 librados: calling monclient init 2024-03-29T11:33:43.636 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.633+0000 7f2dda84a640 10 monclient: init 2024-03-29T11:33:43.636 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.634+0000 7f2dda84a640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:43.636 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.634+0000 7f2dda84a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:33:43.636 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.634+0000 7f2dda84a640 10 monclient(hunting): picked mon.noname-a con 0x7f2dd40a9d70 addr v1:172.21.15.135:6789/0 2024-03-29T11:33:43.636 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.638+0000 7f2dda84a640 10 monclient(hunting): picked mon.noname-c con 0x7f2dd413f650 addr v1:172.21.15.135:6790/0 2024-03-29T11:33:43.637 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.639+0000 7f2dd3fff640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f2dd40a9d70 0x7f2dd4136f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:55969/0 (socket says 172.21.15.135:55969) 2024-03-29T11:33:43.637 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.639+0000 7f2dda84a640 10 monclient(hunting): picked mon.noname-b con 0x7f2dd41457f0 addr v1:172.21.15.194:6789/0 2024-03-29T11:33:43.637 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.639+0000 7f2dda84a640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd40ab970 con 0x7f2dd40a9d70 2024-03-29T11:33:43.637 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.639+0000 7f2dda84a640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd40acb70 con 0x7f2dd41457f0 2024-03-29T11:33:43.637 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.639+0000 7f2dda84a640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd4142db0 con 0x7f2dd413f650 2024-03-29T11:33:43.637 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.639+0000 7f2dda84a640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:43.637 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.639+0000 7f2dda84a640 10 monclient(hunting): authenticate will time out at 1331.008714s 2024-03-29T11:33:43.637 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.639+0000 7f2dd3fff640 1 -- 172.21.15.135:0/2221205376 learned_addr learned my addr 172.21.15.135:0/2221205376 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:33:43.639 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.642+0000 7f2dd0ff9640 10 client.?.objecter ms_handle_connect 0x7f2dd40a9d70 2024-03-29T11:33:43.639 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.642+0000 7f2dd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:43.639 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.642+0000 7f2dd0ff9640 1 -- 172.21.15.135:0/2221205376 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452908561 0 0) 0x7f2dd40ab970 con 0x7f2dd40a9d70 2024-03-29T11:33:43.639 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.642+0000 7f2dd0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:33:43.639 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.642+0000 7f2dd0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:43.639 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.642+0000 7f2dd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:43.639 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.642+0000 7f2dd0ff9640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:33:43.639 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.642+0000 7f2dd0ff9640 1 -- 172.21.15.135:0/2221205376 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dc0004eb0 con 0x7f2dd40a9d70 2024-03-29T11:33:43.640 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.643+0000 7f2dd0ff9640 1 -- 172.21.15.135:0/2221205376 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2946230942 0 0) 0x7f2dc0004eb0 con 0x7f2dd40a9d70 2024-03-29T11:33:43.641 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.643+0000 7f2dd0ff9640 1 -- 172.21.15.135:0/2221205376 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2dd40ab970 con 0x7f2dd40a9d70 2024-03-29T11:33:43.641 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.643+0000 7f2dd0ff9640 1 -- 172.21.15.135:0/2221205376 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f2dc4001970 con 0x7f2dd40a9d70 2024-03-29T11:33:43.641 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.643+0000 7f2dd0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:43.641 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.643+0000 7f2dd0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:33:43.641 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.643+0000 7f2dd0ff9640 10 monclient(hunting): dump: 2024-03-29T11:33:43.641 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:43.641 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:43.641 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:43.641 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:43.641 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:43.641 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:43.641 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:43.641 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:43.641 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:43.642 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:43.642 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.643+0000 7f2dd0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:43.642 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.643+0000 7f2dd0ff9640 10 client.?.objecter ms_handle_connect 0x7f2dd41457f0 2024-03-29T11:33:43.642 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:43.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dd0ff9640 1 -- 172.21.15.135:0/2221205376 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569396530 0 0) 0x7f2dd40acb70 con 0x7f2dd41457f0 2024-03-29T11:33:43.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dd0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:33:43.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dd0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:43.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:43.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dd0ff9640 10 monclient(hunting): my global_id is 4558 2024-03-29T11:33:43.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dd0ff9640 1 -- 172.21.15.135:0/2221205376 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dc0004eb0 con 0x7f2dd41457f0 2024-03-29T11:33:43.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dd0ff9640 1 -- 172.21.15.135:0/2221205376 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1403013916 0 0) 0x7f2dd40ab970 con 0x7f2dd40a9d70 2024-03-29T11:33:43.643 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dd0ff9640 1 -- 172.21.15.135:0/2221205376 >> v1:172.21.15.135:6790/0 conn(0x7f2dd413f650 legacy=0x7f2dd41d74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:43.644 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dd0ff9640 1 -- 172.21.15.135:0/2221205376 >> v1:172.21.15.194:6789/0 conn(0x7f2dd41457f0 legacy=0x7f2dd41e2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:43.644 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dd0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:33:43.644 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dd0ff9640 1 monclient: found mon.a 2024-03-29T11:33:43.644 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dd0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:43.644 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dd0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:33:43.644 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dd0ff9640 1 -- 172.21.15.135:0/2221205376 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd4141f50 con 0x7f2dd40a9d70 2024-03-29T11:33:43.644 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dd0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:33:43.644 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dd0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:33:43.644 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dd0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:43.644 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dda84a640 5 monclient: authenticate success, global_id 4533 2024-03-29T11:33:43.644 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dda84a640 10 monclient: _renew_subs 2024-03-29T11:33:43.644 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dda84a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:33:43.644 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dda84a640 1 -- 172.21.15.135:0/2221205376 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd4137ba0 con 0x7f2dd40a9d70 2024-03-29T11:33:43.644 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dda84a640 10 client.4533.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:43.645 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dda84a640 10 monclient: _renew_subs 2024-03-29T11:33:43.645 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dda84a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:33:43.645 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dda84a640 1 -- 172.21.15.135:0/2221205376 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd4137670 con 0x7f2dd40a9d70 2024-03-29T11:33:43.645 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.644+0000 7f2dda84a640 1 librados: init done 2024-03-29T11:33:43.645 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.645+0000 7f2dd0ff9640 1 -- 172.21.15.135:0/2221205376 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2dc4001c90 con 0x7f2dd40a9d70 2024-03-29T11:33:43.645 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.645+0000 7f2dd0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:43.645 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.645+0000 7f2dd0ff9640 1 -- 172.21.15.135:0/2221205376 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f2dc4003a60 con 0x7f2dd40a9d70 2024-03-29T11:33:43.645 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.645+0000 7f2dd27fc640 4 set_mon_vals no callback set 2024-03-29T11:33:43.645 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.646+0000 7f2dd0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:43.645 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.646+0000 7f2dd0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:33:43.645 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.646+0000 7f2dd0ff9640 10 monclient: dump: 2024-03-29T11:33:43.645 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:43.645 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:43.645 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:43.645 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:43.646 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:43.646 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:43.646 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:43.646 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:43.646 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:43.646 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:43.646 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.648+0000 7f2dda84a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:43.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.648+0000 7f2dda84a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:43.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.648+0000 7f2dda84a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:33:43.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.648+0000 7f2dda84a640 1 -- 172.21.15.135:0/2221205376 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db4005080 con 0x7f2dd40a9d70 2024-03-29T11:33:43.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.649+0000 7f2dd0ff9640 1 -- 172.21.15.135:0/2221205376 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f2dc4003bc0 con 0x7f2dd40a9d70 2024-03-29T11:33:43.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.650+0000 7f2dd0ff9640 1 -- 172.21.15.135:0/2221205376 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f2dc4084b30 con 0x7f2dd40a9d70 2024-03-29T11:33:43.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.650+0000 7f2dd0ff9640 10 client.4533.objecter ms_dispatch 0x7f2dd4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:43.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.650+0000 7f2dd37fe640 10 monclient: get_auth_request con 0x7f2dc006dd90 auth_method 0 2024-03-29T11:33:43.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.650+0000 7f2dd0ff9640 3 client.4533.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:43.647 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.650+0000 7f2dd0ff9640 3 client.4533.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:43.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.650+0000 7f2dd0ff9640 20 client.4533.objecter dump_active .. 0 homeless 2024-03-29T11:33:43.648 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.651+0000 7f2dd0ff9640 10 client.4533.objecter ms_handle_connect 0x7f2dc006dd90 2024-03-29T11:33:43.656 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.659+0000 7f2dd0ff9640 1 -- 172.21.15.135:0/2221205376 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2dc4050410 con 0x7f2dd40a9d70 2024-03-29T11:33:43.656 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.659+0000 7f2dd0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:43.657 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.659+0000 7f2dd0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:43.975 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.978+0000 7f2dda84a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:43.975 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.978+0000 7f2dda84a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:43.975 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.978+0000 7f2dda84a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:33:43.975 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.978+0000 7f2dda84a640 1 -- 172.21.15.135:0/2221205376 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2db4005330 con 0x7f2dd40a9d70 2024-03-29T11:33:43.976 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.979+0000 7f2dd0ff9640 1 -- 172.21.15.135:0/2221205376 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f2dc4052340 con 0x7f2dd40a9d70 2024-03-29T11:33:43.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.979+0000 7f2dd0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:43.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.979+0000 7f2dd0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:43.977 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:33:43.977 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:43.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.983+0000 7f2dda84a640 10 librados: watch_flush enter 2024-03-29T11:33:43.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.983+0000 7f2dda84a640 10 librados: watch_flush exit 2024-03-29T11:33:43.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.983+0000 7f2dda84a640 20 client.4533.objecter shutdown clearing up homeless session... 2024-03-29T11:33:43.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.983+0000 7f2dda84a640 10 client.4533.objecter successfully canceled tick 2024-03-29T11:33:43.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.984+0000 7f2dda84a640 1 -- 172.21.15.135:0/2221205376 >> v1:172.21.15.135:6816/3469875285 conn(0x7f2dc006dd90 legacy=0x7f2dc0070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:43.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.984+0000 7f2dda84a640 10 monclient: shutdown 2024-03-29T11:33:43.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.984+0000 7f2dda84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:43.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.984+0000 7f2dda84a640 1 -- 172.21.15.135:0/2221205376 >> v1:172.21.15.135:6789/0 conn(0x7f2dd40a9d70 legacy=0x7f2dd4136f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:43.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.984+0000 7f2dda84a640 1 -- 172.21.15.135:0/2221205376 shutdown_connections 2024-03-29T11:33:43.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.984+0000 7f2dda84a640 1 -- 172.21.15.135:0/2221205376 >> 172.21.15.135:0/2221205376 conn(0x7f2dd4066a00 msgr2=0x7f2dd40b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:43.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.984+0000 7f2dda84a640 1 -- 172.21.15.135:0/2221205376 shutdown_connections 2024-03-29T11:33:43.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.984+0000 7f2dda84a640 1 -- 172.21.15.135:0/2221205376 wait complete. 2024-03-29T11:33:43.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:43.984+0000 7f2dda84a640 1 librados: shutdown 2024-03-29T11:33:44.003 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:50.810 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:50.975 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.978+0000 7f9ab10a1640 10 monclient: get_monmap_and_config 2024-03-29T11:33:50.975 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.978+0000 7f9ab10a1640 10 monclient: build_initial_monmap 2024-03-29T11:33:50.976 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.978+0000 7f9ab10a1640 10 monclient: monmap: 2024-03-29T11:33:50.976 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:33:50.976 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:50.976 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:33:50.979923+0000 2024-03-29T11:33:50.976 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:33:50.979923+0000 2024-03-29T11:33:50.976 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:50.976 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:50.976 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:33:50.976 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:33:50.976 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:33:50.976 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:50.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.979+0000 7f9ab10a1640 1 Processor -- start 2024-03-29T11:33:50.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.979+0000 7f9ab10a1640 1 -- start start 2024-03-29T11:33:50.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.979+0000 7f9ab10a1640 10 monclient: init 2024-03-29T11:33:50.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.980+0000 7f9ab10a1640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:50.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.980+0000 7f9ab10a1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:33:50.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.980+0000 7f9ab10a1640 10 monclient(hunting): picked mon.noname-b con 0x7f9aac0b0860 addr v1:172.21.15.135:6790/0 2024-03-29T11:33:50.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.980+0000 7f9ab10a1640 10 monclient(hunting): picked mon.noname-c con 0x7f9aac13ef30 addr v1:172.21.15.194:6789/0 2024-03-29T11:33:50.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.980+0000 7f9ab10a1640 10 monclient(hunting): picked mon.noname-a con 0x7f9aac06d9d0 addr v1:172.21.15.135:6789/0 2024-03-29T11:33:50.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.980+0000 7f9ab10a1640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aac0b0c40 con 0x7f9aac06d9d0 2024-03-29T11:33:50.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.980+0000 7f9ab10a1640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aac0aed10 con 0x7f9aac13ef30 2024-03-29T11:33:50.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.980+0000 7f9ab10a1640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aac0ab1b0 con 0x7f9aac0b0860 2024-03-29T11:33:50.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.980+0000 7f9ab10a1640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:50.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.980+0000 7f9ab10a1640 10 monclient(hunting): authenticate will time out at 1338.349938s 2024-03-29T11:33:50.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.981+0000 7f9aaa59c640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f9aac13ef30 0x7f9aac0af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:49230/0 (socket says 172.21.15.135:49230) 2024-03-29T11:33:50.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.981+0000 7f9aaa59c640 1 -- 172.21.15.135:0/992361618 learned_addr learned my addr 172.21.15.135:0/992361618 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:33:50.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.983+0000 7f9aa9d9b640 1 -- 172.21.15.135:0/992361618 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300610813 0 0) 0x7f9aac0b0c40 con 0x7f9aac06d9d0 2024-03-29T11:33:50.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.983+0000 7f9aa9d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:33:50.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.983+0000 7f9aa9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:50.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.983+0000 7f9aa9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:50.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.983+0000 7f9aa9d9b640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:33:50.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.983+0000 7f9aa9d9b640 1 -- 172.21.15.135:0/992361618 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a94003520 con 0x7f9aac06d9d0 2024-03-29T11:33:50.982 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.983+0000 7f9aa9d9b640 1 -- 172.21.15.135:0/992361618 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711052903 0 0) 0x7f9aac0aed10 con 0x7f9aac13ef30 2024-03-29T11:33:50.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.984+0000 7f9aa9d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:33:50.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.984+0000 7f9aa9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:50.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.984+0000 7f9aa9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:50.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.984+0000 7f9aa9d9b640 10 monclient(hunting): my global_id is 4561 2024-03-29T11:33:50.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.984+0000 7f9aa9d9b640 1 -- 172.21.15.135:0/992361618 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aac0b0c40 con 0x7f9aac13ef30 2024-03-29T11:33:50.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.984+0000 7f9aa9d9b640 1 -- 172.21.15.135:0/992361618 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3963571822 0 0) 0x7f9a94003520 con 0x7f9aac06d9d0 2024-03-29T11:33:50.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.984+0000 7f9aa9d9b640 1 -- 172.21.15.135:0/992361618 >> v1:172.21.15.135:6790/0 conn(0x7f9aac0b0860 legacy=0x7f9aac13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:50.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.984+0000 7f9aa9d9b640 1 -- 172.21.15.135:0/992361618 >> v1:172.21.15.194:6789/0 conn(0x7f9aac13ef30 legacy=0x7f9aac0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:50.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.984+0000 7f9aa9d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:33:50.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.984+0000 7f9aa9d9b640 1 monclient: found mon.noname-a 2024-03-29T11:33:50.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.984+0000 7f9aa9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:50.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.984+0000 7f9aa9d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.135:6789/0 2024-03-29T11:33:50.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.984+0000 7f9aa9d9b640 1 -- 172.21.15.135:0/992361618 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aac0ab490 con 0x7f9aac06d9d0 2024-03-29T11:33:50.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.984+0000 7f9aa9d9b640 10 monclient: _finish_auth 0 2024-03-29T11:33:50.983 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.984+0000 7f9aa9d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:33:50.984 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.984+0000 7f9aa9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:50.984 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.984+0000 7f9aa9d9b640 1 -- 172.21.15.135:0/992361618 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f9aa4001a10 con 0x7f9aac06d9d0 2024-03-29T11:33:50.984 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.984+0000 7f9aa9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:50.984 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.984+0000 7f9aa9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:33:50.984 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.984+0000 7f9aa9d9b640 10 monclient: dump: 2024-03-29T11:33:50.984 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:50.984 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:50.984 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:50.984 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:50.984 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:50.984 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:50.984 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:50.984 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:50.984 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:50.984 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:50.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.985+0000 7f9ab10a1640 5 monclient: authenticate success, global_id 4536 2024-03-29T11:33:50.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.985+0000 7f9ab10a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:50.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.987+0000 7f9aa9d9b640 1 -- 172.21.15.135:0/992361618 <== mon.0 v1:172.21.15.135:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9aa4001b70 con 0x7f9aac06d9d0 2024-03-29T11:33:50.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.987+0000 7f9aa9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:50.985 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.987+0000 7f9aa9d9b640 1 -- 172.21.15.135:0/992361618 <== mon.0 v1:172.21.15.135:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f9aa4001f80 con 0x7f9aac06d9d0 2024-03-29T11:33:50.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.987+0000 7f9aa9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:50.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.987+0000 7f9aa9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:33:50.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.987+0000 7f9aa9d9b640 10 monclient: dump: 2024-03-29T11:33:50.986 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:50.986 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:50.986 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:50.986 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:50.986 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:50.986 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:50.986 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:50.986 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:50.986 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:50.986 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:50.986 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.988+0000 7f9ab10a1640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:50.987 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.988+0000 7f9ab10a1640 4 set_mon_vals no callback set 2024-03-29T11:33:50.987 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.989+0000 7f9ab10a1640 10 monclient: shutdown 2024-03-29T11:33:50.987 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.989+0000 7f9ab10a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:50.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.989+0000 7f9ab10a1640 1 -- 172.21.15.135:0/992361618 >> v1:172.21.15.135:6789/0 conn(0x7f9aac06d9d0 legacy=0x7f9aac14a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:50.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.989+0000 7f9ab10a1640 1 -- 172.21.15.135:0/992361618 shutdown_connections 2024-03-29T11:33:50.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.989+0000 7f9ab10a1640 1 -- 172.21.15.135:0/992361618 >> 172.21.15.135:0/992361618 conn(0x7f9aac06b660 msgr2=0x7f9aac06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:50.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.989+0000 7f9ab10a1640 1 -- 172.21.15.135:0/992361618 shutdown_connections 2024-03-29T11:33:50.988 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.989+0000 7f9ab10a1640 1 -- 172.21.15.135:0/992361618 wait complete. 2024-03-29T11:33:50.989 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.991+0000 7f9ab10a1640 10 monclient: build_initial_monmap 2024-03-29T11:33:50.989 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.991+0000 7f9ab10a1640 10 monclient: monmap: 2024-03-29T11:33:50.989 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:33:50.989 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:50.989 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:33:50.989 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:33:50.989 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:50.989 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:50.989 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:33:50.989 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:33:50.989 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:33:50.989 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:50.989 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.991+0000 7f9ab10a1640 1 librados: starting msgr at 2024-03-29T11:33:50.990 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.992+0000 7f9ab10a1640 1 librados: starting objecter 2024-03-29T11:33:50.995 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.992+0000 7f9ab10a1640 1 Processor -- start 2024-03-29T11:33:50.995 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.992+0000 7f9ab10a1640 1 -- start start 2024-03-29T11:33:50.995 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.992+0000 7f9ab10a1640 1 librados: setting wanted keys 2024-03-29T11:33:50.995 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.992+0000 7f9ab10a1640 1 librados: calling monclient init 2024-03-29T11:33:50.995 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.992+0000 7f9ab10a1640 10 monclient: init 2024-03-29T11:33:50.995 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.993+0000 7f9ab10a1640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:50.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.993+0000 7f9ab10a1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:33:50.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.993+0000 7f9ab10a1640 10 monclient(hunting): picked mon.noname-a con 0x7f9aac06d9d0 addr v1:172.21.15.135:6789/0 2024-03-29T11:33:50.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.993+0000 7f9ab10a1640 10 monclient(hunting): picked mon.noname-b con 0x7f9aac0b0860 addr v1:172.21.15.194:6789/0 2024-03-29T11:33:50.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.993+0000 7f9ab10a1640 10 monclient(hunting): picked mon.noname-c con 0x7f9aac13ef30 addr v1:172.21.15.135:6790/0 2024-03-29T11:33:50.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.993+0000 7f9ab10a1640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aac0b40f0 con 0x7f9aac06d9d0 2024-03-29T11:33:50.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.993+0000 7f9ab10a1640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aac06dec0 con 0x7f9aac0b0860 2024-03-29T11:33:50.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.993+0000 7f9ab10a1640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aac0b2bb0 con 0x7f9aac13ef30 2024-03-29T11:33:50.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.993+0000 7f9ab10a1640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:50.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.993+0000 7f9ab10a1640 10 monclient(hunting): authenticate will time out at 1338.362402s 2024-03-29T11:33:50.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.993+0000 7f9aaa59c640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f9aac0b0860 0x7f9aac14b6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:52316/0 (socket says 172.21.15.135:52316) 2024-03-29T11:33:50.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.993+0000 7f9aaa59c640 1 -- 172.21.15.135:0/3418072071 learned_addr learned my addr 172.21.15.135:0/3418072071 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:33:50.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.993+0000 7f9a937fe640 10 client.?.objecter ms_handle_connect 0x7f9aac13ef30 2024-03-29T11:33:50.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.993+0000 7f9a937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:50.996 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.993+0000 7f9a937fe640 10 client.?.objecter ms_handle_connect 0x7f9aac0b0860 2024-03-29T11:33:50.997 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.993+0000 7f9a937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:50.997 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.994+0000 7f9a937fe640 1 -- 172.21.15.135:0/3418072071 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009876902 0 0) 0x7f9aac0b2bb0 con 0x7f9aac13ef30 2024-03-29T11:33:50.997 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.994+0000 7f9a937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:33:50.997 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.994+0000 7f9a937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:50.997 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.994+0000 7f9a937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:50.997 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.994+0000 7f9a937fe640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:33:50.997 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.994+0000 7f9a937fe640 1 -- 172.21.15.135:0/3418072071 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a98004eb0 con 0x7f9aac13ef30 2024-03-29T11:33:50.997 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.994+0000 7f9a937fe640 1 -- 172.21.15.135:0/3418072071 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499254317 0 0) 0x7f9aac06dec0 con 0x7f9aac0b0860 2024-03-29T11:33:50.997 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.994+0000 7f9a937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:33:50.997 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.994+0000 7f9a937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:50.997 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.994+0000 7f9a937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:50.997 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.994+0000 7f9a937fe640 10 monclient(hunting): my global_id is 4564 2024-03-29T11:33:50.997 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.994+0000 7f9a937fe640 1 -- 172.21.15.135:0/3418072071 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aac0b2bb0 con 0x7f9aac0b0860 2024-03-29T11:33:50.997 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.995+0000 7f9a937fe640 1 -- 172.21.15.135:0/3418072071 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1471846968 0 0) 0x7f9aac0b2bb0 con 0x7f9aac0b0860 2024-03-29T11:33:50.998 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.995+0000 7f9a937fe640 1 -- 172.21.15.135:0/3418072071 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9aac06dec0 con 0x7f9aac0b0860 2024-03-29T11:33:50.998 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.995+0000 7f9a937fe640 1 -- 172.21.15.135:0/3418072071 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f9aa0001910 con 0x7f9aac0b0860 2024-03-29T11:33:50.998 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.995+0000 7f9a937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:50.998 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.995+0000 7f9a937fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:33:50.998 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.995+0000 7f9a937fe640 10 monclient(hunting): dump: 2024-03-29T11:33:50.998 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:50.998 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:50.998 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:50.998 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:50.998 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:50.998 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:50.998 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:50.998 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:50.998 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:50.999 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:50.999 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.995+0000 7f9a937fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:50.999 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.995+0000 7f9a937fe640 1 -- 172.21.15.135:0/3418072071 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4256657106 0 0) 0x7f9a98004eb0 con 0x7f9aac13ef30 2024-03-29T11:33:50.999 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.995+0000 7f9a937fe640 1 -- 172.21.15.135:0/3418072071 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9aac0b2bb0 con 0x7f9aac13ef30 2024-03-29T11:33:50.999 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.995+0000 7f9a937fe640 1 -- 172.21.15.135:0/3418072071 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f9aa4001e80 con 0x7f9aac13ef30 2024-03-29T11:33:50.999 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.995+0000 7f9a937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:50.999 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.995+0000 7f9a937fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:50.999 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.995+0000 7f9a937fe640 10 monclient(hunting): dump: 2024-03-29T11:33:50.999 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:50.999 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:50.999 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:50.999 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:50.999 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:50.999 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:50.999 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:51.000 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:51.000 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:51.000 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:51.000 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.995+0000 7f9a937fe640 10 client.?.objecter ms_handle_connect 0x7f9aac06d9d0 2024-03-29T11:33:51.000 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.995+0000 7f9a937fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:51.000 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.995+0000 7f9a937fe640 1 -- 172.21.15.135:0/3418072071 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 973999134 0 0) 0x7f9aac06dec0 con 0x7f9aac0b0860 2024-03-29T11:33:51.000 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.995+0000 7f9a937fe640 1 -- 172.21.15.135:0/3418072071 >> v1:172.21.15.135:6790/0 conn(0x7f9aac13ef30 legacy=0x7f9aac14bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:51.000 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.995+0000 7f9a937fe640 1 -- 172.21.15.135:0/3418072071 >> v1:172.21.15.135:6789/0 conn(0x7f9aac06d9d0 legacy=0x7f9aac1d5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:51.000 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.996+0000 7f9a937fe640 10 monclient: _finish_hunting 0 2024-03-29T11:33:51.000 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.996+0000 7f9a937fe640 1 monclient: found mon.b 2024-03-29T11:33:51.000 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.996+0000 7f9a937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:51.000 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.996+0000 7f9a937fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:33:51.000 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.996+0000 7f9a937fe640 1 -- 172.21.15.135:0/3418072071 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aac14c4f0 con 0x7f9aac0b0860 2024-03-29T11:33:51.000 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.996+0000 7f9a937fe640 10 monclient: _finish_auth 0 2024-03-29T11:33:51.000 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.996+0000 7f9a937fe640 10 monclient: _check_auth_tickets 2024-03-29T11:33:51.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.996+0000 7f9a937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:51.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.996+0000 7f9ab10a1640 5 monclient: authenticate success, global_id 4564 2024-03-29T11:33:51.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.996+0000 7f9ab10a1640 10 monclient: _renew_subs 2024-03-29T11:33:51.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.996+0000 7f9ab10a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:33:51.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.996+0000 7f9ab10a1640 1 -- 172.21.15.135:0/3418072071 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aac0b33b0 con 0x7f9aac0b0860 2024-03-29T11:33:51.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.996+0000 7f9ab10a1640 10 client.4564.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:51.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.996+0000 7f9ab10a1640 10 monclient: _renew_subs 2024-03-29T11:33:51.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.996+0000 7f9ab10a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:33:51.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.996+0000 7f9ab10a1640 1 -- 172.21.15.135:0/3418072071 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aac071b50 con 0x7f9aac0b0860 2024-03-29T11:33:51.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.996+0000 7f9a937fe640 1 -- 172.21.15.135:0/3418072071 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9aa0001a70 con 0x7f9aac0b0860 2024-03-29T11:33:51.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.996+0000 7f9ab10a1640 1 librados: init done 2024-03-29T11:33:51.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.996+0000 7f9a937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:51.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.996+0000 7f9a937fe640 1 -- 172.21.15.135:0/3418072071 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f9aa00049c0 con 0x7f9aac0b0860 2024-03-29T11:33:51.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.996+0000 7f9a937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:51.001 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.996+0000 7f9a937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:33:51.002 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.996+0000 7f9aa959a640 4 set_mon_vals no callback set 2024-03-29T11:33:51.002 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.996+0000 7f9a937fe640 10 monclient: dump: 2024-03-29T11:33:51.002 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:51.002 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:51.002 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:51.002 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:51.002 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:51.002 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:51.002 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:51.002 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:51.002 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:51.002 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:51.003 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.998+0000 7f9a937fe640 1 -- 172.21.15.135:0/3418072071 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f9aa001a990 con 0x7f9aac0b0860 2024-03-29T11:33:51.003 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.999+0000 7f9ab10a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:51.003 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.999+0000 7f9ab10a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:51.003 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.999+0000 7f9ab10a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:33:51.004 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.999+0000 7f9a937fe640 1 -- 172.21.15.135:0/3418072071 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f9aa0084a70 con 0x7f9aac0b0860 2024-03-29T11:33:51.004 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.999+0000 7f9a937fe640 10 client.4564.objecter ms_dispatch 0x7f9aac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:51.004 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.999+0000 7f9a937fe640 3 client.4564.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:51.004 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.999+0000 7f9a937fe640 3 client.4564.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:51.004 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.999+0000 7f9ab10a1640 1 -- 172.21.15.135:0/3418072071 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aac13d0e0 con 0x7f9aac0b0860 2024-03-29T11:33:51.004 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:50.999+0000 7f9a937fe640 20 client.4564.objecter dump_active .. 0 homeless 2024-03-29T11:33:51.004 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:51.000+0000 7f9aaad9d640 10 monclient: get_auth_request con 0x7f9a9806de80 auth_method 0 2024-03-29T11:33:51.004 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:51.000+0000 7f9a937fe640 10 client.4564.objecter ms_handle_connect 0x7f9a9806de80 2024-03-29T11:33:51.004 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:51.006+0000 7f9a937fe640 1 -- 172.21.15.135:0/3418072071 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9aa0050180 con 0x7f9aac0b0860 2024-03-29T11:33:51.004 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:51.006+0000 7f9a937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:51.004 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:51.006+0000 7f9a937fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:51.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:51.354+0000 7f9ab10a1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:51.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:51.354+0000 7f9ab10a1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:51.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:51.354+0000 7f9ab10a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:33:51.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:51.354+0000 7f9ab10a1640 1 -- 172.21.15.135:0/3418072071 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9aac075f10 con 0x7f9aac0b0860 2024-03-29T11:33:51.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:51.354+0000 7f9a937fe640 1 -- 172.21.15.135:0/3418072071 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f9aa0052280 con 0x7f9aac0b0860 2024-03-29T11:33:51.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:51.355+0000 7f9a937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:51.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:51.355+0000 7f9a937fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:51.357 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:33:51.358 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:51.358 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:51.361+0000 7f9ab10a1640 10 librados: watch_flush enter 2024-03-29T11:33:51.359 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:51.361+0000 7f9ab10a1640 10 librados: watch_flush exit 2024-03-29T11:33:51.359 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:51.361+0000 7f9ab10a1640 20 client.4564.objecter shutdown clearing up homeless session... 2024-03-29T11:33:51.359 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:51.361+0000 7f9ab10a1640 10 client.4564.objecter successfully canceled tick 2024-03-29T11:33:51.359 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:51.361+0000 7f9ab10a1640 1 -- 172.21.15.135:0/3418072071 >> v1:172.21.15.135:6816/3469875285 conn(0x7f9a9806de80 legacy=0x7f9a98070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:51.359 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:51.361+0000 7f9ab10a1640 10 monclient: shutdown 2024-03-29T11:33:51.359 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:51.361+0000 7f9ab10a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:51.359 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:51.361+0000 7f9ab10a1640 1 -- 172.21.15.135:0/3418072071 >> v1:172.21.15.194:6789/0 conn(0x7f9aac0b0860 legacy=0x7f9aac14b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:51.359 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:51.361+0000 7f9ab10a1640 1 -- 172.21.15.135:0/3418072071 shutdown_connections 2024-03-29T11:33:51.359 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:51.361+0000 7f9ab10a1640 1 -- 172.21.15.135:0/3418072071 >> 172.21.15.135:0/3418072071 conn(0x7f9aac06b660 msgr2=0x7f9aac13e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:51.360 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:51.362+0000 7f9ab10a1640 1 -- 172.21.15.135:0/3418072071 shutdown_connections 2024-03-29T11:33:51.360 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:51.362+0000 7f9ab10a1640 1 -- 172.21.15.135:0/3418072071 wait complete. 2024-03-29T11:33:51.361 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:51.364+0000 7f9ab10a1640 1 librados: shutdown 2024-03-29T11:33:51.383 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:33:58.190 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:33:58.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.355+0000 7f8f4763e640 10 monclient: get_monmap_and_config 2024-03-29T11:33:58.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.355+0000 7f8f4763e640 10 monclient: build_initial_monmap 2024-03-29T11:33:58.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.355+0000 7f8f4763e640 10 monclient: monmap: 2024-03-29T11:33:58.352 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:33:58.352 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:58.352 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:33:58.355897+0000 2024-03-29T11:33:58.352 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:33:58.355897+0000 2024-03-29T11:33:58.352 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:58.352 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:58.352 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:33:58.352 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:33:58.352 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:33:58.352 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:58.353 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.356+0000 7f8f4763e640 1 Processor -- start 2024-03-29T11:33:58.353 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.356+0000 7f8f4763e640 1 -- start start 2024-03-29T11:33:58.353 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.356+0000 7f8f4763e640 10 monclient: init 2024-03-29T11:33:58.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.357+0000 7f8f4763e640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:58.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.357+0000 7f8f4763e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:33:58.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.357+0000 7f8f4763e640 10 monclient(hunting): picked mon.noname-c con 0x7f8f401422b0 addr v1:172.21.15.194:6789/0 2024-03-29T11:33:58.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.357+0000 7f8f4763e640 10 monclient(hunting): picked mon.noname-a con 0x7f8f40142690 addr v1:172.21.15.135:6789/0 2024-03-29T11:33:58.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.357+0000 7f8f4763e640 10 monclient(hunting): picked mon.noname-b con 0x7f8f400b17d0 addr v1:172.21.15.135:6790/0 2024-03-29T11:33:58.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.357+0000 7f8f4763e640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f40142c80 con 0x7f8f40142690 2024-03-29T11:33:58.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.357+0000 7f8f4763e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f400aae40 con 0x7f8f401422b0 2024-03-29T11:33:58.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.357+0000 7f8f4763e640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f4013fbd0 con 0x7f8f400b17d0 2024-03-29T11:33:58.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.357+0000 7f8f4763e640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:58.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.357+0000 7f8f4763e640 10 monclient(hunting): authenticate will time out at 1345.725667s 2024-03-29T11:33:58.355 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.357+0000 7f8f45bdb640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f8f400b17d0 0x7f8f400b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:43052/0 (socket says 172.21.15.135:43052) 2024-03-29T11:33:58.355 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.357+0000 7f8f45bdb640 1 -- 172.21.15.135:0/2601035451 learned_addr learned my addr 172.21.15.135:0/2601035451 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:33:58.356 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.358+0000 7f8f33fff640 1 -- 172.21.15.135:0/2601035451 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4233007760 0 0) 0x7f8f4013fbd0 con 0x7f8f400b17d0 2024-03-29T11:33:58.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.358+0000 7f8f33fff640 10 monclient(hunting): _negotiate 2024-03-29T11:33:58.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.358+0000 7f8f33fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:58.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.358+0000 7f8f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:58.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.358+0000 7f8f33fff640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:33:58.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.358+0000 7f8f33fff640 1 -- 172.21.15.135:0/2601035451 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f2c003520 con 0x7f8f400b17d0 2024-03-29T11:33:58.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.358+0000 7f8f33fff640 1 -- 172.21.15.135:0/2601035451 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318578030 0 0) 0x7f8f400aae40 con 0x7f8f401422b0 2024-03-29T11:33:58.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.358+0000 7f8f33fff640 10 monclient(hunting): _negotiate 2024-03-29T11:33:58.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.358+0000 7f8f33fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:58.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.358+0000 7f8f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:58.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.358+0000 7f8f33fff640 10 monclient(hunting): my global_id is 4567 2024-03-29T11:33:58.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.358+0000 7f8f33fff640 1 -- 172.21.15.135:0/2601035451 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f4013fbd0 con 0x7f8f401422b0 2024-03-29T11:33:58.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.359+0000 7f8f33fff640 1 -- 172.21.15.135:0/2601035451 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3288608440 0 0) 0x7f8f2c003520 con 0x7f8f400b17d0 2024-03-29T11:33:58.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.359+0000 7f8f33fff640 1 -- 172.21.15.135:0/2601035451 >> v1:172.21.15.194:6789/0 conn(0x7f8f401422b0 legacy=0x7f8f4013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:58.358 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.359+0000 7f8f33fff640 1 -- 172.21.15.135:0/2601035451 >> v1:172.21.15.135:6789/0 conn(0x7f8f40142690 legacy=0x7f8f400b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:58.358 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.359+0000 7f8f33fff640 10 monclient: _finish_hunting 0 2024-03-29T11:33:58.358 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.359+0000 7f8f33fff640 1 monclient: found mon.noname-b 2024-03-29T11:33:58.358 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.359+0000 7f8f33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:58.358 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.359+0000 7f8f33fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:33:58.358 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.359+0000 7f8f33fff640 1 -- 172.21.15.135:0/2601035451 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f40151940 con 0x7f8f400b17d0 2024-03-29T11:33:58.358 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.359+0000 7f8f33fff640 10 monclient: _finish_auth 0 2024-03-29T11:33:58.358 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.359+0000 7f8f33fff640 10 monclient: _check_auth_tickets 2024-03-29T11:33:58.358 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.359+0000 7f8f33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:58.358 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.359+0000 7f8f33fff640 1 -- 172.21.15.135:0/2601035451 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f8f38001a10 con 0x7f8f400b17d0 2024-03-29T11:33:58.358 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.359+0000 7f8f33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:58.358 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.359+0000 7f8f33fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:33:58.358 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.359+0000 7f8f33fff640 10 monclient: dump: 2024-03-29T11:33:58.358 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:58.358 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:58.359 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:58.359 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:58.359 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:58.359 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:58.359 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:58.359 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:58.359 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:58.359 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:58.359 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.359+0000 7f8f4763e640 5 monclient: authenticate success, global_id 4586 2024-03-29T11:33:58.359 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.359+0000 7f8f4763e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:33:58.359 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.359+0000 7f8f33fff640 1 -- 172.21.15.135:0/2601035451 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f38001b70 con 0x7f8f400b17d0 2024-03-29T11:33:58.359 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.359+0000 7f8f33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:58.359 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.359+0000 7f8f33fff640 1 -- 172.21.15.135:0/2601035451 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f8f38001f00 con 0x7f8f400b17d0 2024-03-29T11:33:58.360 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.359+0000 7f8f33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:58.360 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.360+0000 7f8f33fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:33:58.360 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.360+0000 7f8f33fff640 10 monclient: dump: 2024-03-29T11:33:58.360 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:58.360 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:58.360 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:58.360 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:58.360 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:58.360 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:58.360 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:58.360 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:58.360 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:58.360 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:58.360 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.360+0000 7f8f4763e640 10 monclient: get_monmap_and_config success 2024-03-29T11:33:58.360 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.360+0000 7f8f4763e640 4 set_mon_vals no callback set 2024-03-29T11:33:58.363 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.360+0000 7f8f4763e640 10 monclient: shutdown 2024-03-29T11:33:58.363 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.360+0000 7f8f4763e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:58.363 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.361+0000 7f8f4763e640 1 -- 172.21.15.135:0/2601035451 >> v1:172.21.15.135:6790/0 conn(0x7f8f400b17d0 legacy=0x7f8f400b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:58.363 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.361+0000 7f8f4763e640 1 -- 172.21.15.135:0/2601035451 shutdown_connections 2024-03-29T11:33:58.364 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.361+0000 7f8f4763e640 1 -- 172.21.15.135:0/2601035451 >> 172.21.15.135:0/2601035451 conn(0x7f8f40066a00 msgr2=0x7f8f40066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:58.364 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.361+0000 7f8f4763e640 1 -- 172.21.15.135:0/2601035451 shutdown_connections 2024-03-29T11:33:58.364 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.361+0000 7f8f4763e640 1 -- 172.21.15.135:0/2601035451 wait complete. 2024-03-29T11:33:58.364 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.362+0000 7f8f4763e640 10 monclient: build_initial_monmap 2024-03-29T11:33:58.364 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.362+0000 7f8f4763e640 10 monclient: monmap: 2024-03-29T11:33:58.364 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:33:58.364 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:33:58.364 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:33:58.364 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:33:58.364 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:33:58.364 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:58.364 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:33:58.364 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:33:58.365 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:33:58.365 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:58.365 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.363+0000 7f8f4763e640 1 librados: starting msgr at 2024-03-29T11:33:58.365 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.363+0000 7f8f4763e640 1 librados: starting objecter 2024-03-29T11:33:58.365 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.363+0000 7f8f4763e640 1 Processor -- start 2024-03-29T11:33:58.365 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.363+0000 7f8f4763e640 1 -- start start 2024-03-29T11:33:58.365 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.363+0000 7f8f4763e640 1 librados: setting wanted keys 2024-03-29T11:33:58.365 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.363+0000 7f8f4763e640 1 librados: calling monclient init 2024-03-29T11:33:58.365 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.363+0000 7f8f4763e640 10 monclient: init 2024-03-29T11:33:58.365 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.364+0000 7f8f4763e640 10 monclient: _reopen_session rank -1 2024-03-29T11:33:58.365 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.364+0000 7f8f4763e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:33:58.365 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.364+0000 7f8f4763e640 10 monclient(hunting): picked mon.noname-b con 0x7f8f400b17d0 addr v1:172.21.15.194:6789/0 2024-03-29T11:33:58.365 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.364+0000 7f8f4763e640 10 monclient(hunting): picked mon.noname-a con 0x7f8f401422b0 addr v1:172.21.15.135:6789/0 2024-03-29T11:33:58.365 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.364+0000 7f8f4763e640 10 monclient(hunting): picked mon.noname-c con 0x7f8f40142690 addr v1:172.21.15.135:6790/0 2024-03-29T11:33:58.365 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.364+0000 7f8f4763e640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f400ac590 con 0x7f8f401422b0 2024-03-29T11:33:58.366 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.364+0000 7f8f453da640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f8f400b17d0 0x7f8f40133c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:60508/0 (socket says 172.21.15.135:60508) 2024-03-29T11:33:58.366 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.364+0000 7f8f4763e640 1 -- 172.21.15.135:0/4136686370 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f400b1230 con 0x7f8f400b17d0 2024-03-29T11:33:58.366 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.364+0000 7f8f4763e640 1 -- 172.21.15.135:0/4136686370 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f400b1460 con 0x7f8f40142690 2024-03-29T11:33:58.366 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.364+0000 7f8f453da640 1 -- 172.21.15.135:0/4136686370 learned_addr learned my addr 172.21.15.135:0/4136686370 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:33:58.366 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.364+0000 7f8f4763e640 10 monclient(hunting): _renew_subs 2024-03-29T11:33:58.366 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.364+0000 7f8f4763e640 10 monclient(hunting): authenticate will time out at 1345.732966s 2024-03-29T11:33:58.366 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.365+0000 7f8f31ffb640 10 client.?.objecter ms_handle_connect 0x7f8f400b17d0 2024-03-29T11:33:58.369 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.365+0000 7f8f31ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:58.370 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.365+0000 7f8f31ffb640 10 client.?.objecter ms_handle_connect 0x7f8f40142690 2024-03-29T11:33:58.370 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.365+0000 7f8f31ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:58.370 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.365+0000 7f8f31ffb640 10 client.?.objecter ms_handle_connect 0x7f8f401422b0 2024-03-29T11:33:58.370 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.365+0000 7f8f31ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:33:58.370 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.365+0000 7f8f31ffb640 1 -- 172.21.15.135:0/4136686370 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 426585686 0 0) 0x7f8f400b1460 con 0x7f8f40142690 2024-03-29T11:33:58.370 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.365+0000 7f8f31ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:33:58.370 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.365+0000 7f8f31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:58.370 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.365+0000 7f8f31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:58.370 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.365+0000 7f8f31ffb640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:33:58.370 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.365+0000 7f8f31ffb640 1 -- 172.21.15.135:0/4136686370 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f24004eb0 con 0x7f8f40142690 2024-03-29T11:33:58.370 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.365+0000 7f8f31ffb640 1 -- 172.21.15.135:0/4136686370 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 957306730 0 0) 0x7f8f400b1230 con 0x7f8f400b17d0 2024-03-29T11:33:58.370 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.365+0000 7f8f31ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:33:58.371 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.365+0000 7f8f31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:58.371 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.365+0000 7f8f31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:58.371 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.365+0000 7f8f31ffb640 10 monclient(hunting): my global_id is 4570 2024-03-29T11:33:58.371 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.365+0000 7f8f31ffb640 1 -- 172.21.15.135:0/4136686370 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f400b1460 con 0x7f8f400b17d0 2024-03-29T11:33:58.371 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.365+0000 7f8f31ffb640 1 -- 172.21.15.135:0/4136686370 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989664971 0 0) 0x7f8f400ac590 con 0x7f8f401422b0 2024-03-29T11:33:58.371 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.365+0000 7f8f31ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:33:58.371 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.365+0000 7f8f31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:33:58.371 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.365+0000 7f8f31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:33:58.371 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.365+0000 7f8f31ffb640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:33:58.371 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.365+0000 7f8f31ffb640 1 -- 172.21.15.135:0/4136686370 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f400b1230 con 0x7f8f401422b0 2024-03-29T11:33:58.371 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f31ffb640 1 -- 172.21.15.135:0/4136686370 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1554738812 0 0) 0x7f8f24004eb0 con 0x7f8f40142690 2024-03-29T11:33:58.371 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f31ffb640 1 -- 172.21.15.135:0/4136686370 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f400ac590 con 0x7f8f40142690 2024-03-29T11:33:58.371 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f31ffb640 1 -- 172.21.15.135:0/4136686370 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f8f380018d0 con 0x7f8f40142690 2024-03-29T11:33:58.372 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f31ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:58.372 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f31ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:33:58.372 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f31ffb640 10 monclient(hunting): dump: 2024-03-29T11:33:58.372 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:58.372 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:58.372 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:58.372 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:58.372 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:58.372 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:58.372 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:58.372 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:58.372 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:58.372 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:58.372 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f31ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:33:58.373 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f31ffb640 1 -- 172.21.15.135:0/4136686370 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2546835052 0 0) 0x7f8f400b1460 con 0x7f8f400b17d0 2024-03-29T11:33:58.373 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f31ffb640 1 -- 172.21.15.135:0/4136686370 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f24004eb0 con 0x7f8f400b17d0 2024-03-29T11:33:58.373 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f31ffb640 1 -- 172.21.15.135:0/4136686370 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f8f340019a0 con 0x7f8f400b17d0 2024-03-29T11:33:58.373 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f31ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:58.373 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f31ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:33:58.373 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f31ffb640 10 monclient(hunting): dump: 2024-03-29T11:33:58.373 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:58.373 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:58.373 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:58.373 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:58.373 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:58.373 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:58.373 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:58.373 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:58.374 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:58.374 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:58.374 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f31ffb640 1 -- 172.21.15.135:0/4136686370 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2813914428 0 0) 0x7f8f24004eb0 con 0x7f8f400b17d0 2024-03-29T11:33:58.374 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f31ffb640 1 -- 172.21.15.135:0/4136686370 >> v1:172.21.15.135:6790/0 conn(0x7f8f40142690 legacy=0x7f8f401e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:58.374 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f31ffb640 1 -- 172.21.15.135:0/4136686370 >> v1:172.21.15.135:6789/0 conn(0x7f8f401422b0 legacy=0x7f8f400af860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:58.374 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f31ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:33:58.374 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f31ffb640 1 monclient: found mon.b 2024-03-29T11:33:58.374 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f31ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:33:58.374 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f31ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:33:58.374 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f31ffb640 1 -- 172.21.15.135:0/4136686370 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f400af340 con 0x7f8f400b17d0 2024-03-29T11:33:58.374 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f31ffb640 10 monclient: _finish_auth 0 2024-03-29T11:33:58.374 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f31ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:33:58.374 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:33:58.374 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f4763e640 5 monclient: authenticate success, global_id 4570 2024-03-29T11:33:58.374 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f4763e640 10 monclient: _renew_subs 2024-03-29T11:33:58.375 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f4763e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:33:58.375 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.366+0000 7f8f4763e640 1 -- 172.21.15.135:0/4136686370 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f4013c430 con 0x7f8f400b17d0 2024-03-29T11:33:58.375 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.367+0000 7f8f4763e640 10 client.4570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:33:58.375 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.367+0000 7f8f4763e640 10 monclient: _renew_subs 2024-03-29T11:33:58.375 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.367+0000 7f8f4763e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:33:58.375 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.367+0000 7f8f4763e640 1 -- 172.21.15.135:0/4136686370 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f400aef70 con 0x7f8f400b17d0 2024-03-29T11:33:58.375 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.367+0000 7f8f4763e640 1 librados: init done 2024-03-29T11:33:58.376 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.367+0000 7f8f31ffb640 1 -- 172.21.15.135:0/4136686370 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f34003610 con 0x7f8f400b17d0 2024-03-29T11:33:58.376 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.367+0000 7f8f31ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:33:58.376 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.367+0000 7f8f31ffb640 1 -- 172.21.15.135:0/4136686370 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f8f340039a0 con 0x7f8f400b17d0 2024-03-29T11:33:58.376 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.367+0000 7f8f31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:33:58.377 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.367+0000 7f8f31ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:33:58.377 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.367+0000 7f8f31ffb640 10 monclient: dump: 2024-03-29T11:33:58.377 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:33:58.377 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:33:58.377 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:58.377 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:33:58.377 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:33:58.377 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:33:58.377 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:33:58.377 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:33:58.377 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:33:58.377 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:33:58.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.367+0000 7f8f337fe640 4 set_mon_vals no callback set 2024-03-29T11:33:58.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.368+0000 7f8f4763e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:58.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.370+0000 7f8f4763e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:58.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.370+0000 7f8f4763e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:33:58.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.370+0000 7f8f4763e640 1 -- 172.21.15.135:0/4136686370 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f20005080 con 0x7f8f400b17d0 2024-03-29T11:33:58.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.370+0000 7f8f31ffb640 1 -- 172.21.15.135:0/4136686370 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f8f34003b80 con 0x7f8f400b17d0 2024-03-29T11:33:58.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.372+0000 7f8f44bd9640 10 monclient: get_auth_request con 0x7f8f2406df00 auth_method 0 2024-03-29T11:33:58.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.372+0000 7f8f31ffb640 1 -- 172.21.15.135:0/4136686370 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f8f34084a90 con 0x7f8f400b17d0 2024-03-29T11:33:58.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.372+0000 7f8f31ffb640 10 client.4570.objecter ms_dispatch 0x7f8f40000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:33:58.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.372+0000 7f8f31ffb640 3 client.4570.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:33:58.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.372+0000 7f8f31ffb640 3 client.4570.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:33:58.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.372+0000 7f8f31ffb640 20 client.4570.objecter dump_active .. 0 homeless 2024-03-29T11:33:58.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.372+0000 7f8f31ffb640 10 client.4570.objecter ms_handle_connect 0x7f8f2406df00 2024-03-29T11:33:58.378 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.376+0000 7f8f31ffb640 1 -- 172.21.15.135:0/4136686370 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8f34050370 con 0x7f8f400b17d0 2024-03-29T11:33:58.379 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.377+0000 7f8f31ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:33:58.379 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.377+0000 7f8f31ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:33:58.795 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.799+0000 7f8f4763e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:58.796 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.799+0000 7f8f4763e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:58.796 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.799+0000 7f8f4763e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:33:58.796 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.799+0000 7f8f4763e640 1 -- 172.21.15.135:0/4136686370 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f20005330 con 0x7f8f400b17d0 2024-03-29T11:33:58.796 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.800+0000 7f8f31ffb640 1 -- 172.21.15.135:0/4136686370 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f8f340522a0 con 0x7f8f400b17d0 2024-03-29T11:33:58.796 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.800+0000 7f8f31ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:33:58.797 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.800+0000 7f8f31ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:33:58.797 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:33:58.797 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:33:58.801 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.804+0000 7f8f4763e640 10 librados: watch_flush enter 2024-03-29T11:33:58.801 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.804+0000 7f8f4763e640 10 librados: watch_flush exit 2024-03-29T11:33:58.801 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.804+0000 7f8f4763e640 20 client.4570.objecter shutdown clearing up homeless session... 2024-03-29T11:33:58.801 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.804+0000 7f8f4763e640 10 client.4570.objecter successfully canceled tick 2024-03-29T11:33:58.801 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.804+0000 7f8f4763e640 1 -- 172.21.15.135:0/4136686370 >> v1:172.21.15.135:6816/3469875285 conn(0x7f8f2406df00 legacy=0x7f8f240703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:58.801 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.804+0000 7f8f4763e640 10 monclient: shutdown 2024-03-29T11:33:58.801 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.804+0000 7f8f4763e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:33:58.802 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.804+0000 7f8f4763e640 1 -- 172.21.15.135:0/4136686370 >> v1:172.21.15.194:6789/0 conn(0x7f8f400b17d0 legacy=0x7f8f40133c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:33:58.802 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.804+0000 7f8f4763e640 1 -- 172.21.15.135:0/4136686370 shutdown_connections 2024-03-29T11:33:58.802 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.804+0000 7f8f4763e640 1 -- 172.21.15.135:0/4136686370 >> 172.21.15.135:0/4136686370 conn(0x7f8f40066a00 msgr2=0x7f8f4013d650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:33:58.802 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.805+0000 7f8f4763e640 1 -- 172.21.15.135:0/4136686370 shutdown_connections 2024-03-29T11:33:58.802 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.805+0000 7f8f4763e640 1 -- 172.21.15.135:0/4136686370 wait complete. 2024-03-29T11:33:58.802 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:33:58.805+0000 7f8f4763e640 1 librados: shutdown 2024-03-29T11:33:58.818 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:05.625 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:05.770 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.770+0000 7f1a5000f640 10 monclient: get_monmap_and_config 2024-03-29T11:34:05.770 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.770+0000 7f1a5000f640 10 monclient: build_initial_monmap 2024-03-29T11:34:05.770 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.770+0000 7f1a5000f640 10 monclient: monmap: 2024-03-29T11:34:05.770 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:34:05.771 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:05.771 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:34:05.771403+0000 2024-03-29T11:34:05.771 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:34:05.771403+0000 2024-03-29T11:34:05.771 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:05.771 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:05.771 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:34:05.771 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:34:05.771 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:34:05.771 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:05.771 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.771+0000 7f1a5000f640 1 Processor -- start 2024-03-29T11:34:05.771 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.772+0000 7f1a5000f640 1 -- start start 2024-03-29T11:34:05.771 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.772+0000 7f1a5000f640 10 monclient: init 2024-03-29T11:34:05.771 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.772+0000 7f1a5000f640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:05.771 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.772+0000 7f1a5000f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:05.772 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.772+0000 7f1a5000f640 10 monclient(hunting): picked mon.noname-a con 0x7f1a480ad6c0 addr v1:172.21.15.135:6789/0 2024-03-29T11:34:05.772 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.772+0000 7f1a5000f640 10 monclient(hunting): picked mon.noname-b con 0x7f1a480af6c0 addr v1:172.21.15.135:6790/0 2024-03-29T11:34:05.772 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.772+0000 7f1a5000f640 10 monclient(hunting): picked mon.noname-c con 0x7f1a4814fdc0 addr v1:172.21.15.194:6789/0 2024-03-29T11:34:05.772 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.772+0000 7f1a5000f640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a480a94e0 con 0x7f1a480ad6c0 2024-03-29T11:34:05.772 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.772+0000 7f1a5000f640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a480af3a0 con 0x7f1a4814fdc0 2024-03-29T11:34:05.772 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.772+0000 7f1a5000f640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a480afb60 con 0x7f1a480af6c0 2024-03-29T11:34:05.772 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.773+0000 7f1a5000f640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:05.772 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.773+0000 7f1a5000f640 10 monclient(hunting): authenticate will time out at 1353.141048s 2024-03-29T11:34:05.773 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.775+0000 7f1a4d5aa640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f1a480af6c0 0x7f1a48152980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:44772/0 (socket says 172.21.15.135:44772) 2024-03-29T11:34:05.773 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.775+0000 7f1a4d5aa640 1 -- 172.21.15.135:0/3964911802 learned_addr learned my addr 172.21.15.135:0/3964911802 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:34:05.773 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.776+0000 7f1a4cda9640 1 -- 172.21.15.135:0/3964911802 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512720441 0 0) 0x7f1a480afb60 con 0x7f1a480af6c0 2024-03-29T11:34:05.773 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.776+0000 7f1a4cda9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:05.773 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.776+0000 7f1a4cda9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:05.773 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.776+0000 7f1a4cda9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:05.773 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.776+0000 7f1a4cda9640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:34:05.773 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.776+0000 7f1a4cda9640 1 -- 172.21.15.135:0/3964911802 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a34003520 con 0x7f1a480af6c0 2024-03-29T11:34:05.776 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.777+0000 7f1a4cda9640 1 -- 172.21.15.135:0/3964911802 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 618659611 0 0) 0x7f1a34003520 con 0x7f1a480af6c0 2024-03-29T11:34:05.776 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.777+0000 7f1a4cda9640 1 -- 172.21.15.135:0/3964911802 >> v1:172.21.15.194:6789/0 conn(0x7f1a4814fdc0 legacy=0x7f1a4815a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:05.776 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.777+0000 7f1a4cda9640 1 -- 172.21.15.135:0/3964911802 >> v1:172.21.15.135:6789/0 conn(0x7f1a480ad6c0 legacy=0x7f1a4814f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:05.776 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.777+0000 7f1a4cda9640 10 monclient: _finish_hunting 0 2024-03-29T11:34:05.776 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.777+0000 7f1a4cda9640 1 monclient: found mon.noname-b 2024-03-29T11:34:05.776 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.777+0000 7f1a4cda9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:05.776 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.777+0000 7f1a4cda9640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:34:05.776 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.777+0000 7f1a4cda9640 1 -- 172.21.15.135:0/3964911802 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a480aab80 con 0x7f1a480af6c0 2024-03-29T11:34:05.776 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.777+0000 7f1a4cda9640 10 monclient: _finish_auth 0 2024-03-29T11:34:05.776 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.777+0000 7f1a4cda9640 10 monclient: _check_auth_tickets 2024-03-29T11:34:05.777 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.777+0000 7f1a4cda9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:05.777 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.777+0000 7f1a4cda9640 1 -- 172.21.15.135:0/3964911802 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f1a40001670 con 0x7f1a480af6c0 2024-03-29T11:34:05.777 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.777+0000 7f1a4cda9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:05.777 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.777+0000 7f1a4cda9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:05.777 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.777+0000 7f1a4cda9640 10 monclient: dump: 2024-03-29T11:34:05.777 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:05.777 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:05.777 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:05.777 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:05.777 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:05.777 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:05.777 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:05.777 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:05.777 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:05.777 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:05.778 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.777+0000 7f1a4cda9640 1 -- 172.21.15.135:0/3964911802 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a40001840 con 0x7f1a480af6c0 2024-03-29T11:34:05.778 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.777+0000 7f1a4cda9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:05.778 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.777+0000 7f1a4cda9640 1 -- 172.21.15.135:0/3964911802 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f1a40001c50 con 0x7f1a480af6c0 2024-03-29T11:34:05.778 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.777+0000 7f1a4cda9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:05.778 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.777+0000 7f1a4cda9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:05.778 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.777+0000 7f1a4cda9640 10 monclient: dump: 2024-03-29T11:34:05.778 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:05.778 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:05.778 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:05.778 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:05.778 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:05.778 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:05.778 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:05.779 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:05.779 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:05.779 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:05.779 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.778+0000 7f1a5000f640 5 monclient: authenticate success, global_id 4592 2024-03-29T11:34:05.779 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.778+0000 7f1a5000f640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:05.779 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.778+0000 7f1a5000f640 4 set_mon_vals no callback set 2024-03-29T11:34:05.779 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.779+0000 7f1a5000f640 10 monclient: shutdown 2024-03-29T11:34:05.779 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.779+0000 7f1a5000f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:05.779 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.779+0000 7f1a5000f640 1 -- 172.21.15.135:0/3964911802 >> v1:172.21.15.135:6790/0 conn(0x7f1a480af6c0 legacy=0x7f1a48152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:05.779 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.779+0000 7f1a5000f640 1 -- 172.21.15.135:0/3964911802 shutdown_connections 2024-03-29T11:34:05.779 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.779+0000 7f1a5000f640 1 -- 172.21.15.135:0/3964911802 >> 172.21.15.135:0/3964911802 conn(0x7f1a480b35e0 msgr2=0x7f1a480b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:05.779 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.779+0000 7f1a5000f640 1 -- 172.21.15.135:0/3964911802 shutdown_connections 2024-03-29T11:34:05.779 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.779+0000 7f1a5000f640 1 -- 172.21.15.135:0/3964911802 wait complete. 2024-03-29T11:34:05.780 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.780+0000 7f1a5000f640 10 monclient: build_initial_monmap 2024-03-29T11:34:05.781 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.780+0000 7f1a5000f640 10 monclient: monmap: 2024-03-29T11:34:05.781 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:34:05.781 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:05.781 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:34:05.781 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:34:05.781 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:05.781 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:05.781 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:34:05.781 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:34:05.781 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:34:05.781 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:05.781 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.780+0000 7f1a5000f640 1 librados: starting msgr at 2024-03-29T11:34:05.781 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.780+0000 7f1a5000f640 1 librados: starting objecter 2024-03-29T11:34:05.781 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.781+0000 7f1a5000f640 1 Processor -- start 2024-03-29T11:34:05.782 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.781+0000 7f1a5000f640 1 -- start start 2024-03-29T11:34:05.782 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.781+0000 7f1a5000f640 1 librados: setting wanted keys 2024-03-29T11:34:05.782 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.781+0000 7f1a5000f640 1 librados: calling monclient init 2024-03-29T11:34:05.782 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.781+0000 7f1a5000f640 10 monclient: init 2024-03-29T11:34:05.782 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.782+0000 7f1a5000f640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:05.782 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.782+0000 7f1a5000f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:05.782 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.782+0000 7f1a5000f640 10 monclient(hunting): picked mon.noname-c con 0x7f1a480ad6c0 addr v1:172.21.15.135:6790/0 2024-03-29T11:34:05.782 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.782+0000 7f1a5000f640 10 monclient(hunting): picked mon.noname-a con 0x7f1a480af6c0 addr v1:172.21.15.135:6789/0 2024-03-29T11:34:05.782 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.782+0000 7f1a5000f640 10 monclient(hunting): picked mon.noname-b con 0x7f1a4814fdc0 addr v1:172.21.15.194:6789/0 2024-03-29T11:34:05.782 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.782+0000 7f1a5000f640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a48071f80 con 0x7f1a480af6c0 2024-03-29T11:34:05.782 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.782+0000 7f1a5000f640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a4806f940 con 0x7f1a4814fdc0 2024-03-29T11:34:05.782 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.782+0000 7f1a5000f640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a48070480 con 0x7f1a480ad6c0 2024-03-29T11:34:05.782 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.782+0000 7f1a5000f640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:05.783 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.782+0000 7f1a5000f640 10 monclient(hunting): authenticate will time out at 1353.150201s 2024-03-29T11:34:05.787 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.785+0000 7f1a4d5aa640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f1a480af6c0 0x7f1a481eca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:33155/0 (socket says 172.21.15.135:33155) 2024-03-29T11:34:05.788 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.785+0000 7f1a4d5aa640 1 -- 172.21.15.135:0/196425736 learned_addr learned my addr 172.21.15.135:0/196425736 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:34:05.788 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.785+0000 7f1a3e7fc640 10 client.?.objecter ms_handle_connect 0x7f1a480af6c0 2024-03-29T11:34:05.788 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.785+0000 7f1a3e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:05.788 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.785+0000 7f1a3e7fc640 1 -- 172.21.15.135:0/196425736 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 694471770 0 0) 0x7f1a48071f80 con 0x7f1a480af6c0 2024-03-29T11:34:05.788 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.785+0000 7f1a3e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:05.788 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.785+0000 7f1a3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:05.788 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.785+0000 7f1a3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:05.788 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.785+0000 7f1a3e7fc640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:34:05.788 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.785+0000 7f1a3e7fc640 1 -- 172.21.15.135:0/196425736 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a2c004eb0 con 0x7f1a480af6c0 2024-03-29T11:34:05.788 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.785+0000 7f1a3e7fc640 1 -- 172.21.15.135:0/196425736 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 226812352 0 0) 0x7f1a2c004eb0 con 0x7f1a480af6c0 2024-03-29T11:34:05.788 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.785+0000 7f1a3e7fc640 1 -- 172.21.15.135:0/196425736 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a2c005300 con 0x7f1a480af6c0 2024-03-29T11:34:05.788 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.786+0000 7f1a3e7fc640 1 -- 172.21.15.135:0/196425736 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f1a40001ff0 con 0x7f1a480af6c0 2024-03-29T11:34:05.788 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.786+0000 7f1a3e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:05.789 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.786+0000 7f1a3e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:05.789 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.786+0000 7f1a3e7fc640 10 monclient(hunting): dump: 2024-03-29T11:34:05.789 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:05.789 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:05.789 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:05.789 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:05.789 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:05.789 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:05.789 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:05.789 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:05.789 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:05.789 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:05.789 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.786+0000 7f1a3e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:05.789 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.786+0000 7f1a3e7fc640 1 -- 172.21.15.135:0/196425736 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 27662749 0 0) 0x7f1a2c005300 con 0x7f1a480af6c0 2024-03-29T11:34:05.789 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.786+0000 7f1a3e7fc640 1 -- 172.21.15.135:0/196425736 >> v1:172.21.15.135:6790/0 conn(0x7f1a480ad6c0 legacy=0x7f1a481e9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:05.790 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.786+0000 7f1a3e7fc640 1 -- 172.21.15.135:0/196425736 >> v1:172.21.15.194:6789/0 conn(0x7f1a4814fdc0 legacy=0x7f1a481f82d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:34:05.790 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.787+0000 7f1a3e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:34:05.790 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.787+0000 7f1a3e7fc640 1 monclient: found mon.a 2024-03-29T11:34:05.790 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.787+0000 7f1a3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:05.790 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.787+0000 7f1a3e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:34:05.790 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.787+0000 7f1a3e7fc640 1 -- 172.21.15.135:0/196425736 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a48070610 con 0x7f1a480af6c0 2024-03-29T11:34:05.790 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.787+0000 7f1a3e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:34:05.790 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.787+0000 7f1a3e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:34:05.790 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.787+0000 7f1a3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:05.790 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.787+0000 7f1a3e7fc640 10 client.?.objecter ms_handle_connect 0x7f1a480ad6c0 2024-03-29T11:34:05.790 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.787+0000 7f1a3e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:05.790 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.787+0000 7f1a5000f640 5 monclient: authenticate success, global_id 4545 2024-03-29T11:34:05.790 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.787+0000 7f1a5000f640 10 monclient: _renew_subs 2024-03-29T11:34:05.791 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.787+0000 7f1a5000f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:34:05.791 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.787+0000 7f1a5000f640 1 -- 172.21.15.135:0/196425736 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a48070d80 con 0x7f1a480af6c0 2024-03-29T11:34:05.791 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.787+0000 7f1a5000f640 10 client.4545.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:05.791 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.787+0000 7f1a5000f640 10 monclient: _renew_subs 2024-03-29T11:34:05.791 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.787+0000 7f1a5000f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:34:05.791 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.787+0000 7f1a5000f640 1 -- 172.21.15.135:0/196425736 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a481ea5b0 con 0x7f1a480af6c0 2024-03-29T11:34:05.791 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.787+0000 7f1a5000f640 1 librados: init done 2024-03-29T11:34:05.791 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.788+0000 7f1a3e7fc640 1 -- 172.21.15.135:0/196425736 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a400016c0 con 0x7f1a480af6c0 2024-03-29T11:34:05.791 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.788+0000 7f1a3e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:05.791 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.788+0000 7f1a3e7fc640 1 -- 172.21.15.135:0/196425736 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f1a40001ff0 con 0x7f1a480af6c0 2024-03-29T11:34:05.791 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.788+0000 7f1a3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:05.791 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.788+0000 7f1a3e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:05.791 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.788+0000 7f1a3e7fc640 10 monclient: dump: 2024-03-29T11:34:05.791 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:05.791 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:05.792 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:05.792 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:05.792 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:05.792 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:05.792 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:05.792 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:05.792 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:05.792 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:05.792 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.788+0000 7f1a3ffff640 4 set_mon_vals no callback set 2024-03-29T11:34:05.792 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.790+0000 7f1a3e7fc640 1 -- 172.21.15.135:0/196425736 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f1a4001ae20 con 0x7f1a480af6c0 2024-03-29T11:34:05.792 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.790+0000 7f1a5000f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:05.792 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.790+0000 7f1a5000f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:05.792 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.790+0000 7f1a5000f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:34:05.792 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.790+0000 7f1a5000f640 1 -- 172.21.15.135:0/196425736 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a30005080 con 0x7f1a480af6c0 2024-03-29T11:34:05.792 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.791+0000 7f1a3e7fc640 1 -- 172.21.15.135:0/196425736 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7f1a40084c60 con 0x7f1a480af6c0 2024-03-29T11:34:05.793 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.791+0000 7f1a3e7fc640 10 client.4545.objecter ms_dispatch 0x7f1a48000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:05.793 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.791+0000 7f1a3e7fc640 3 client.4545.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:05.793 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.791+0000 7f1a3e7fc640 3 client.4545.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:05.793 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.791+0000 7f1a3e7fc640 20 client.4545.objecter dump_active .. 0 homeless 2024-03-29T11:34:05.793 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.793+0000 7f1a4ddab640 10 monclient: get_auth_request con 0x7f1a2c06df30 auth_method 0 2024-03-29T11:34:05.793 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.794+0000 7f1a3e7fc640 10 client.4545.objecter ms_handle_connect 0x7f1a2c06df30 2024-03-29T11:34:05.796 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.800+0000 7f1a3e7fc640 1 -- 172.21.15.135:0/196425736 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1a400505a0 con 0x7f1a480af6c0 2024-03-29T11:34:05.796 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.800+0000 7f1a3e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:05.797 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:05.800+0000 7f1a3e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:06.126 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:06.129+0000 7f1a5000f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:06.126 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:06.129+0000 7f1a5000f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:06.126 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:06.129+0000 7f1a5000f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:34:06.126 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:06.129+0000 7f1a5000f640 1 -- 172.21.15.135:0/196425736 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a30005330 con 0x7f1a480af6c0 2024-03-29T11:34:06.127 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:06.130+0000 7f1a3e7fc640 1 -- 172.21.15.135:0/196425736 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7f1a400505a0 con 0x7f1a480af6c0 2024-03-29T11:34:06.127 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:06.130+0000 7f1a3e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:06.128 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:06.130+0000 7f1a3e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:06.128 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:34:06.128 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:06.131 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:06.134+0000 7f1a5000f640 10 librados: watch_flush enter 2024-03-29T11:34:06.131 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:06.134+0000 7f1a5000f640 10 librados: watch_flush exit 2024-03-29T11:34:06.132 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:06.134+0000 7f1a5000f640 20 client.4545.objecter shutdown clearing up homeless session... 2024-03-29T11:34:06.132 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:06.134+0000 7f1a5000f640 10 client.4545.objecter successfully canceled tick 2024-03-29T11:34:06.132 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:06.134+0000 7f1a5000f640 1 -- 172.21.15.135:0/196425736 >> v1:172.21.15.135:6816/3469875285 conn(0x7f1a2c06df30 legacy=0x7f1a2c0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:06.132 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:06.134+0000 7f1a5000f640 10 monclient: shutdown 2024-03-29T11:34:06.132 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:06.134+0000 7f1a5000f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:06.132 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:06.135+0000 7f1a5000f640 1 -- 172.21.15.135:0/196425736 >> v1:172.21.15.135:6789/0 conn(0x7f1a480af6c0 legacy=0x7f1a481eca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:06.132 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:06.135+0000 7f1a5000f640 1 -- 172.21.15.135:0/196425736 shutdown_connections 2024-03-29T11:34:06.132 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:06.135+0000 7f1a5000f640 1 -- 172.21.15.135:0/196425736 >> 172.21.15.135:0/196425736 conn(0x7f1a480b35e0 msgr2=0x7f1a4814f480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:06.132 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:06.135+0000 7f1a5000f640 1 -- 172.21.15.135:0/196425736 shutdown_connections 2024-03-29T11:34:06.132 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:06.135+0000 7f1a5000f640 1 -- 172.21.15.135:0/196425736 wait complete. 2024-03-29T11:34:06.132 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:06.135+0000 7f1a5000f640 1 librados: shutdown 2024-03-29T11:34:06.151 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:12.958 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:13.107 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.110+0000 7fadc078f640 10 monclient: get_monmap_and_config 2024-03-29T11:34:13.107 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.110+0000 7fadc078f640 10 monclient: build_initial_monmap 2024-03-29T11:34:13.107 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.110+0000 7fadc078f640 10 monclient: monmap: 2024-03-29T11:34:13.107 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:34:13.107 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:13.107 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:34:13.111092+0000 2024-03-29T11:34:13.107 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:34:13.111092+0000 2024-03-29T11:34:13.107 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:13.107 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:13.107 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:34:13.107 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:34:13.107 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:34:13.107 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:13.111 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.111+0000 7fadc078f640 1 Processor -- start 2024-03-29T11:34:13.111 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.111+0000 7fadc078f640 1 -- start start 2024-03-29T11:34:13.111 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.111+0000 7fadc078f640 10 monclient: init 2024-03-29T11:34:13.112 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.112+0000 7fadc078f640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:13.112 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.112+0000 7fadc078f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:13.112 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.112+0000 7fadc078f640 10 monclient(hunting): picked mon.noname-c con 0x7fadb8070120 addr v1:172.21.15.194:6789/0 2024-03-29T11:34:13.112 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.112+0000 7fadc078f640 10 monclient(hunting): picked mon.noname-b con 0x7fadb80b0420 addr v1:172.21.15.135:6790/0 2024-03-29T11:34:13.112 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.112+0000 7fadc078f640 10 monclient(hunting): picked mon.noname-a con 0x7fadb8070500 addr v1:172.21.15.135:6789/0 2024-03-29T11:34:13.112 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.112+0000 7fadc078f640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadb8072090 con 0x7fadb8070500 2024-03-29T11:34:13.112 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.112+0000 7fadc078f640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadb8072370 con 0x7fadb8070120 2024-03-29T11:34:13.112 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.112+0000 7fadc078f640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadb8072650 con 0x7fadb80b0420 2024-03-29T11:34:13.112 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.112+0000 7fadc078f640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:13.112 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.112+0000 7fadc078f640 10 monclient(hunting): authenticate will time out at 1360.480582s 2024-03-29T11:34:13.112 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.114+0000 7fadbdd2a640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7fadb80b0420 0x7fadb81405b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:43090/0 (socket says 172.21.15.135:43090) 2024-03-29T11:34:13.112 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.114+0000 7fadbdd2a640 1 -- 172.21.15.135:0/3825734322 learned_addr learned my addr 172.21.15.135:0/3825734322 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:34:13.114 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.117+0000 7fadbd529640 1 -- 172.21.15.135:0/3825734322 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2609508910 0 0) 0x7fadb8072650 con 0x7fadb80b0420 2024-03-29T11:34:13.114 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.117+0000 7fadbd529640 10 monclient(hunting): _negotiate 2024-03-29T11:34:13.114 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.117+0000 7fadbd529640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:13.114 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.117+0000 7fadbd529640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:13.114 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.117+0000 7fadbd529640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:34:13.114 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.117+0000 7fadbd529640 1 -- 172.21.15.135:0/3825734322 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fada4003520 con 0x7fadb80b0420 2024-03-29T11:34:13.115 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadbd529640 1 -- 172.21.15.135:0/3825734322 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314688875 0 0) 0x7fadb8072370 con 0x7fadb8070120 2024-03-29T11:34:13.115 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadbd529640 10 monclient(hunting): _negotiate 2024-03-29T11:34:13.115 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadbd529640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:13.115 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadbd529640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:13.115 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadbd529640 10 monclient(hunting): my global_id is 4573 2024-03-29T11:34:13.116 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadbd529640 1 -- 172.21.15.135:0/3825734322 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadb8072650 con 0x7fadb8070120 2024-03-29T11:34:13.116 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadbd529640 1 -- 172.21.15.135:0/3825734322 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4248564246 0 0) 0x7fada4003520 con 0x7fadb80b0420 2024-03-29T11:34:13.116 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadbd529640 1 -- 172.21.15.135:0/3825734322 >> v1:172.21.15.194:6789/0 conn(0x7fadb8070120 legacy=0x7fadb813d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:13.116 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadbd529640 1 -- 172.21.15.135:0/3825734322 >> v1:172.21.15.135:6789/0 conn(0x7fadb8070500 legacy=0x7fadb814b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:13.116 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadbd529640 10 monclient: _finish_hunting 0 2024-03-29T11:34:13.116 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadbd529640 1 monclient: found mon.noname-b 2024-03-29T11:34:13.116 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadbd529640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:13.116 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadbd529640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.135:6790/0 2024-03-29T11:34:13.116 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadbd529640 1 -- 172.21.15.135:0/3825734322 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb8140090 con 0x7fadb80b0420 2024-03-29T11:34:13.116 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadbd529640 10 monclient: _finish_auth 0 2024-03-29T11:34:13.116 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadbd529640 10 monclient: _check_auth_tickets 2024-03-29T11:34:13.116 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadbd529640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:13.116 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadc078f640 5 monclient: authenticate success, global_id 4595 2024-03-29T11:34:13.116 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadc078f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:13.117 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadbd529640 1 -- 172.21.15.135:0/3825734322 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fadb0001a10 con 0x7fadb80b0420 2024-03-29T11:34:13.117 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadbd529640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:13.117 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadbd529640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:13.117 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadbd529640 10 monclient: dump: 2024-03-29T11:34:13.117 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:13.117 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:13.117 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:13.117 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:13.117 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:13.117 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:13.117 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:13.117 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:13.117 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:13.117 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:13.118 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadc078f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:13.118 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadbd529640 1 -- 172.21.15.135:0/3825734322 <== mon.2 v1:172.21.15.135:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fadb0001b70 con 0x7fadb80b0420 2024-03-29T11:34:13.118 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadbd529640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:13.118 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadc078f640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:13.118 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.118+0000 7fadc078f640 4 set_mon_vals no callback set 2024-03-29T11:34:13.118 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.119+0000 7fadbd529640 1 -- 172.21.15.135:0/3825734322 <== mon.2 v1:172.21.15.135:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fadb0001f00 con 0x7fadb80b0420 2024-03-29T11:34:13.118 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.119+0000 7fadbd529640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:13.118 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.119+0000 7fadbd529640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:13.118 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.119+0000 7fadbd529640 10 monclient: dump: 2024-03-29T11:34:13.118 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:13.118 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:13.118 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:13.118 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:13.118 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:13.119 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:13.119 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:13.119 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:13.119 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:13.119 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:13.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.119+0000 7fadc078f640 10 monclient: shutdown 2024-03-29T11:34:13.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.119+0000 7fadc078f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:13.119 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.119+0000 7fadc078f640 1 -- 172.21.15.135:0/3825734322 >> v1:172.21.15.135:6790/0 conn(0x7fadb80b0420 legacy=0x7fadb81405b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:13.120 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.120+0000 7fadc078f640 1 -- 172.21.15.135:0/3825734322 shutdown_connections 2024-03-29T11:34:13.120 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.120+0000 7fadc078f640 1 -- 172.21.15.135:0/3825734322 >> 172.21.15.135:0/3825734322 conn(0x7fadb8067500 msgr2=0x7fadb80b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:13.120 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.120+0000 7fadc078f640 1 -- 172.21.15.135:0/3825734322 shutdown_connections 2024-03-29T11:34:13.120 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.120+0000 7fadc078f640 1 -- 172.21.15.135:0/3825734322 wait complete. 2024-03-29T11:34:13.120 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.121+0000 7fadc078f640 10 monclient: build_initial_monmap 2024-03-29T11:34:13.120 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.121+0000 7fadc078f640 10 monclient: monmap: 2024-03-29T11:34:13.120 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:34:13.120 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:13.120 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:34:13.120 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:34:13.120 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:13.120 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:13.120 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:34:13.120 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:34:13.120 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:34:13.120 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:13.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.121+0000 7fadc078f640 1 librados: starting msgr at 2024-03-29T11:34:13.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.121+0000 7fadc078f640 1 librados: starting objecter 2024-03-29T11:34:13.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.122+0000 7fadc078f640 1 Processor -- start 2024-03-29T11:34:13.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.122+0000 7fadc078f640 1 -- start start 2024-03-29T11:34:13.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.122+0000 7fadc078f640 1 librados: setting wanted keys 2024-03-29T11:34:13.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.122+0000 7fadc078f640 1 librados: calling monclient init 2024-03-29T11:34:13.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.122+0000 7fadc078f640 10 monclient: init 2024-03-29T11:34:13.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.122+0000 7fadc078f640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:13.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.122+0000 7fadc078f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:34:13.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.122+0000 7fadc078f640 10 monclient(hunting): picked mon.noname-a con 0x7fadb8070120 addr v1:172.21.15.135:6789/0 2024-03-29T11:34:13.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.122+0000 7fadc078f640 10 monclient(hunting): picked mon.noname-b con 0x7fadb8070500 addr v1:172.21.15.194:6789/0 2024-03-29T11:34:13.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.123+0000 7fadbed2c640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7fadb80b0420 0x7fadb8131d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:44810/0 (socket says 172.21.15.135:44810) 2024-03-29T11:34:13.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.123+0000 7fadbed2c640 1 -- 172.21.15.135:0/1494108423 learned_addr learned my addr 172.21.15.135:0/1494108423 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:34:13.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.123+0000 7fadc078f640 10 monclient(hunting): picked mon.noname-c con 0x7fadb80b0420 addr v1:172.21.15.135:6790/0 2024-03-29T11:34:13.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.123+0000 7fadc078f640 1 -- 172.21.15.135:0/1494108423 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadb80a9e10 con 0x7fadb8070120 2024-03-29T11:34:13.121 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.123+0000 7fadc078f640 1 -- 172.21.15.135:0/1494108423 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadb80aa060 con 0x7fadb8070500 2024-03-29T11:34:13.122 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.123+0000 7fadc078f640 1 -- 172.21.15.135:0/1494108423 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadb80aa340 con 0x7fadb80b0420 2024-03-29T11:34:13.122 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.123+0000 7fadaeffd640 10 client.?.objecter ms_handle_connect 0x7fadb80b0420 2024-03-29T11:34:13.122 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.123+0000 7fadaeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:13.122 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.123+0000 7fadc078f640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:13.122 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.123+0000 7fadc078f640 10 monclient(hunting): authenticate will time out at 1360.491316s 2024-03-29T11:34:13.122 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.123+0000 7fadaeffd640 1 -- 172.21.15.135:0/1494108423 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1872375790 0 0) 0x7fadb80aa340 con 0x7fadb80b0420 2024-03-29T11:34:13.122 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.123+0000 7fadaeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:13.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.123+0000 7fadaeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:13.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.123+0000 7fadaeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:13.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.123+0000 7fadaeffd640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:34:13.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.123+0000 7fadaeffd640 1 -- 172.21.15.135:0/1494108423 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad9c004eb0 con 0x7fadb80b0420 2024-03-29T11:34:13.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.123+0000 7fadaeffd640 1 -- 172.21.15.135:0/1494108423 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1205562350 0 0) 0x7fad9c004eb0 con 0x7fadb80b0420 2024-03-29T11:34:13.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.123+0000 7fadaeffd640 1 -- 172.21.15.135:0/1494108423 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadb80aa340 con 0x7fadb80b0420 2024-03-29T11:34:13.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.123+0000 7fadaeffd640 1 -- 172.21.15.135:0/1494108423 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fada80018e0 con 0x7fadb80b0420 2024-03-29T11:34:13.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.123+0000 7fadaeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:13.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.123+0000 7fadaeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:13.123 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.123+0000 7fadaeffd640 10 monclient(hunting): dump: 2024-03-29T11:34:13.123 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:13.123 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:13.123 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:13.123 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:13.123 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:13.124 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:13.124 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:13.124 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:13.124 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:13.124 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:13.124 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.123+0000 7fadaeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:13.124 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.123+0000 7fadaeffd640 1 -- 172.21.15.135:0/1494108423 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3408792158 0 0) 0x7fadb80aa340 con 0x7fadb80b0420 2024-03-29T11:34:13.124 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.124+0000 7fadaeffd640 1 -- 172.21.15.135:0/1494108423 >> v1:172.21.15.194:6789/0 conn(0x7fadb8070500 legacy=0x7fadb80aa7d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:34:13.125 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.128+0000 7fadaeffd640 1 -- 172.21.15.135:0/1494108423 >> v1:172.21.15.135:6789/0 conn(0x7fadb8070120 legacy=0x7fadb8139a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:13.125 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.128+0000 7fadaeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:13.125 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.128+0000 7fadaeffd640 1 monclient: found mon.c 2024-03-29T11:34:13.125 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.128+0000 7fadaeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:13.125 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.128+0000 7fadaeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:34:13.125 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.129+0000 7fadaeffd640 1 -- 172.21.15.135:0/1494108423 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb813a260 con 0x7fadb80b0420 2024-03-29T11:34:13.125 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.129+0000 7fadaeffd640 10 monclient: _finish_auth 0 2024-03-29T11:34:13.125 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.129+0000 7fadaeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:13.126 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.129+0000 7fadaeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:13.126 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.129+0000 7fadc078f640 5 monclient: authenticate success, global_id 4598 2024-03-29T11:34:13.126 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.129+0000 7fadc078f640 10 monclient: _renew_subs 2024-03-29T11:34:13.126 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.129+0000 7fadc078f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:34:13.126 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.129+0000 7fadc078f640 1 -- 172.21.15.135:0/1494108423 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadb8132420 con 0x7fadb80b0420 2024-03-29T11:34:13.126 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.129+0000 7fadaeffd640 10 client.4598.objecter ms_handle_connect 0x7fadb8070120 2024-03-29T11:34:13.126 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.129+0000 7fadaeffd640 10 client.4598.objecter resend_mon_ops 2024-03-29T11:34:13.126 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.129+0000 7fadc078f640 10 client.4598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:13.126 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.129+0000 7fadc078f640 10 monclient: _renew_subs 2024-03-29T11:34:13.127 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.129+0000 7fadc078f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:34:13.127 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.129+0000 7fadc078f640 1 -- 172.21.15.135:0/1494108423 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fadb8139080 con 0x7fadb80b0420 2024-03-29T11:34:13.127 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.129+0000 7fadc078f640 1 librados: init done 2024-03-29T11:34:13.129 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.131+0000 7fadaeffd640 1 -- 172.21.15.135:0/1494108423 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fada8001d30 con 0x7fadb80b0420 2024-03-29T11:34:13.130 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.131+0000 7fadaeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:13.130 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.131+0000 7fadbcd28640 4 set_mon_vals no callback set 2024-03-29T11:34:13.130 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.132+0000 7fadaeffd640 1 -- 172.21.15.135:0/1494108423 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fada80038a0 con 0x7fadb80b0420 2024-03-29T11:34:13.130 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.132+0000 7fadaeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:13.130 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.132+0000 7fadaeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:13.130 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.132+0000 7fadaeffd640 10 monclient: dump: 2024-03-29T11:34:13.130 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:13.130 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:13.130 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:13.130 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:13.130 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:13.130 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:13.130 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:13.130 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:13.130 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:13.131 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:13.131 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.132+0000 7fadaeffd640 1 -- 172.21.15.135:0/1494108423 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fada8003ac0 con 0x7fadb80b0420 2024-03-29T11:34:13.131 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.134+0000 7fadc078f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:13.131 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.134+0000 7fadc078f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:13.131 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.134+0000 7fadc078f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:34:13.131 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.134+0000 7fadc078f640 1 -- 172.21.15.135:0/1494108423 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada0005080 con 0x7fadb80b0420 2024-03-29T11:34:13.135 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.134+0000 7fadaeffd640 1 -- 172.21.15.135:0/1494108423 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7fada8084ac0 con 0x7fadb80b0420 2024-03-29T11:34:13.135 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.134+0000 7fadaeffd640 10 client.4598.objecter ms_dispatch 0x7fadb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:13.135 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.134+0000 7fadaeffd640 3 client.4598.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:13.135 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.134+0000 7fadaeffd640 3 client.4598.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:13.135 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.134+0000 7fadaeffd640 20 client.4598.objecter dump_active .. 0 homeless 2024-03-29T11:34:13.135 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.134+0000 7fadbe52b640 10 monclient: get_auth_request con 0x7fad9c06dc10 auth_method 0 2024-03-29T11:34:13.136 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.134+0000 7fadaeffd640 10 client.4598.objecter ms_handle_connect 0x7fad9c06dc10 2024-03-29T11:34:13.140 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.144+0000 7fadaeffd640 1 -- 172.21.15.135:0/1494108423 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fada8050420 con 0x7fadb80b0420 2024-03-29T11:34:13.140 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.144+0000 7fadaeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:13.140 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.144+0000 7fadaeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:13.462 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.465+0000 7fadc078f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:13.462 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.465+0000 7fadc078f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:13.462 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.465+0000 7fadc078f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:34:13.463 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.465+0000 7fadc078f640 1 -- 172.21.15.135:0/1494108423 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fada0005330 con 0x7fadb80b0420 2024-03-29T11:34:13.466 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:34:13.466 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:13.467 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.467+0000 7fadaeffd640 1 -- 172.21.15.135:0/1494108423 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7fada8052350 con 0x7fadb80b0420 2024-03-29T11:34:13.467 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.467+0000 7fadaeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:13.467 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.467+0000 7fadaeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:13.468 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.470+0000 7fadc078f640 10 librados: watch_flush enter 2024-03-29T11:34:13.468 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.471+0000 7fadc078f640 10 librados: watch_flush exit 2024-03-29T11:34:13.468 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.471+0000 7fadc078f640 20 client.4598.objecter shutdown clearing up homeless session... 2024-03-29T11:34:13.468 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.471+0000 7fadc078f640 10 client.4598.objecter successfully canceled tick 2024-03-29T11:34:13.468 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.471+0000 7fadc078f640 1 -- 172.21.15.135:0/1494108423 >> v1:172.21.15.135:6816/3469875285 conn(0x7fad9c06dc10 legacy=0x7fad9c0700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:13.468 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.471+0000 7fadc078f640 10 monclient: shutdown 2024-03-29T11:34:13.468 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.471+0000 7fadc078f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:13.468 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.471+0000 7fadc078f640 1 -- 172.21.15.135:0/1494108423 >> v1:172.21.15.135:6790/0 conn(0x7fadb80b0420 legacy=0x7fadb8131d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:13.468 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.471+0000 7fadc078f640 1 -- 172.21.15.135:0/1494108423 shutdown_connections 2024-03-29T11:34:13.468 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.471+0000 7fadc078f640 1 -- 172.21.15.135:0/1494108423 >> 172.21.15.135:0/1494108423 conn(0x7fadb8067500 msgr2=0x7fadb81405b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:13.469 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.472+0000 7fadc078f640 1 -- 172.21.15.135:0/1494108423 shutdown_connections 2024-03-29T11:34:13.470 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.473+0000 7fadc078f640 1 -- 172.21.15.135:0/1494108423 wait complete. 2024-03-29T11:34:13.470 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:13.474+0000 7fadc078f640 1 librados: shutdown 2024-03-29T11:34:13.488 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:20.296 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:20.447 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.449+0000 7fead7fff640 10 monclient: get_monmap_and_config 2024-03-29T11:34:20.447 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.449+0000 7fead7fff640 10 monclient: build_initial_monmap 2024-03-29T11:34:20.447 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.449+0000 7fead7fff640 10 monclient: monmap: 2024-03-29T11:34:20.447 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:34:20.447 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:20.447 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:34:20.450851+0000 2024-03-29T11:34:20.447 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:34:20.450851+0000 2024-03-29T11:34:20.447 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:20.448 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:20.448 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:34:20.448 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:34:20.448 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:34:20.448 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:20.448 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.450+0000 7fead7fff640 1 Processor -- start 2024-03-29T11:34:20.448 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.450+0000 7fead7fff640 1 -- start start 2024-03-29T11:34:20.449 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.450+0000 7fead7fff640 10 monclient: init 2024-03-29T11:34:20.452 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.451+0000 7fead7fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:20.452 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.451+0000 7fead7fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:20.452 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.451+0000 7fead7fff640 10 monclient(hunting): picked mon.noname-c con 0x7fead80b2800 addr v1:172.21.15.194:6789/0 2024-03-29T11:34:20.452 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.451+0000 7fead7fff640 10 monclient(hunting): picked mon.noname-b con 0x7fead80a9c00 addr v1:172.21.15.135:6790/0 2024-03-29T11:34:20.452 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.451+0000 7fead7fff640 10 monclient(hunting): picked mon.noname-a con 0x7fead80b38c0 addr v1:172.21.15.135:6789/0 2024-03-29T11:34:20.452 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.451+0000 7fead7fff640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fead8141eb0 con 0x7fead80b38c0 2024-03-29T11:34:20.452 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.451+0000 7fead7fff640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fead80b22d0 con 0x7fead80b2800 2024-03-29T11:34:20.452 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.451+0000 7fead7fff640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fead8142d30 con 0x7fead80a9c00 2024-03-29T11:34:20.453 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.451+0000 7fead7fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:20.453 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.451+0000 7fead7fff640 10 monclient(hunting): authenticate will time out at 1367.820514s 2024-03-29T11:34:20.453 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.454+0000 7fead77fe640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fead80b38c0 0x7fead814d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:46929/0 (socket says 172.21.15.135:46929) 2024-03-29T11:34:20.453 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.454+0000 7fead77fe640 1 -- 172.21.15.135:0/1935655489 learned_addr learned my addr 172.21.15.135:0/1935655489 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:34:20.454 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.455+0000 7fead67fc640 1 -- 172.21.15.135:0/1935655489 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 396083985 0 0) 0x7fead8141eb0 con 0x7fead80b38c0 2024-03-29T11:34:20.454 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.455+0000 7fead67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:34:20.454 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.455+0000 7fead67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:20.454 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.455+0000 7fead67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:20.454 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.455+0000 7fead67fc640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:34:20.454 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.455+0000 7fead67fc640 1 -- 172.21.15.135:0/1935655489 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feac4003520 con 0x7fead80b38c0 2024-03-29T11:34:20.454 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.455+0000 7fead67fc640 1 -- 172.21.15.135:0/1935655489 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2652209993 0 0) 0x7feac4003520 con 0x7fead80b38c0 2024-03-29T11:34:20.454 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.455+0000 7fead67fc640 1 -- 172.21.15.135:0/1935655489 >> v1:172.21.15.135:6790/0 conn(0x7fead80a9c00 legacy=0x7fead813bb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:34:20.457 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.457+0000 7fead67fc640 1 -- 172.21.15.135:0/1935655489 >> v1:172.21.15.194:6789/0 conn(0x7fead80b2800 legacy=0x7fead8142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:34:20.457 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.457+0000 7fead67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:34:20.457 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.457+0000 7fead67fc640 1 monclient: found mon.noname-a 2024-03-29T11:34:20.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.457+0000 7fead67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:20.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.457+0000 7fead67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.135:6789/0 2024-03-29T11:34:20.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.457+0000 7fead67fc640 1 -- 172.21.15.135:0/1935655489 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead80aaf90 con 0x7fead80b38c0 2024-03-29T11:34:20.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.457+0000 7fead67fc640 10 monclient: _finish_auth 0 2024-03-29T11:34:20.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.457+0000 7fead67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:34:20.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.457+0000 7fead67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:20.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.457+0000 7fead67fc640 1 -- 172.21.15.135:0/1935655489 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fead0001a10 con 0x7fead80b38c0 2024-03-29T11:34:20.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.457+0000 7fead67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:20.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.457+0000 7fead67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:20.458 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.457+0000 7fead67fc640 10 monclient: dump: 2024-03-29T11:34:20.458 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:20.458 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:20.458 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:20.459 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:20.459 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:20.459 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:20.459 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:20.459 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:20.459 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:20.459 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:20.459 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.458+0000 7fead67fc640 1 -- 172.21.15.135:0/1935655489 <== mon.0 v1:172.21.15.135:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fead0001c30 con 0x7fead80b38c0 2024-03-29T11:34:20.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.458+0000 7fead67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:20.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.458+0000 7fead67fc640 1 -- 172.21.15.135:0/1935655489 <== mon.0 v1:172.21.15.135:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fead0001fc0 con 0x7fead80b38c0 2024-03-29T11:34:20.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.458+0000 7fead67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:20.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.458+0000 7fead67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:20.460 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.458+0000 7fead67fc640 10 monclient: dump: 2024-03-29T11:34:20.460 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:20.460 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:20.460 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:20.460 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:20.460 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:20.460 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:20.460 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:20.460 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:20.461 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:20.461 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:20.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.458+0000 7fead7fff640 5 monclient: authenticate success, global_id 4551 2024-03-29T11:34:20.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.458+0000 7fead7fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:20.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.458+0000 7fead7fff640 4 set_mon_vals no callback set 2024-03-29T11:34:20.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.459+0000 7fead7fff640 10 monclient: shutdown 2024-03-29T11:34:20.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.459+0000 7fead7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:20.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.459+0000 7fead7fff640 1 -- 172.21.15.135:0/1935655489 >> v1:172.21.15.135:6789/0 conn(0x7fead80b38c0 legacy=0x7fead814d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:20.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.460+0000 7fead7fff640 1 -- 172.21.15.135:0/1935655489 shutdown_connections 2024-03-29T11:34:20.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.460+0000 7fead7fff640 1 -- 172.21.15.135:0/1935655489 >> 172.21.15.135:0/1935655489 conn(0x7fead8066a00 msgr2=0x7fead8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:20.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.460+0000 7fead7fff640 1 -- 172.21.15.135:0/1935655489 shutdown_connections 2024-03-29T11:34:20.461 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.460+0000 7fead7fff640 1 -- 172.21.15.135:0/1935655489 wait complete. 2024-03-29T11:34:20.462 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.461+0000 7fead7fff640 10 monclient: build_initial_monmap 2024-03-29T11:34:20.463 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.461+0000 7fead7fff640 10 monclient: monmap: 2024-03-29T11:34:20.463 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:34:20.463 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:20.463 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:34:20.463 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:34:20.463 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:20.463 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:20.463 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:34:20.463 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:34:20.463 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:34:20.463 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:20.463 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.461+0000 7fead7fff640 1 librados: starting msgr at 2024-03-29T11:34:20.463 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.461+0000 7fead7fff640 1 librados: starting objecter 2024-03-29T11:34:20.463 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.461+0000 7fead7fff640 1 Processor -- start 2024-03-29T11:34:20.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.464+0000 7fead7fff640 1 -- start start 2024-03-29T11:34:20.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.464+0000 7fead7fff640 1 librados: setting wanted keys 2024-03-29T11:34:20.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.464+0000 7fead7fff640 1 librados: calling monclient init 2024-03-29T11:34:20.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.464+0000 7fead7fff640 10 monclient: init 2024-03-29T11:34:20.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.465+0000 7fead7fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:20.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.465+0000 7fead7fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:34:20.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.465+0000 7fead7fff640 10 monclient(hunting): picked mon.noname-b con 0x7fead80a9c00 addr v1:172.21.15.194:6789/0 2024-03-29T11:34:20.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.465+0000 7fead7fff640 10 monclient(hunting): picked mon.noname-a con 0x7fead80b2800 addr v1:172.21.15.135:6789/0 2024-03-29T11:34:20.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.465+0000 7fead7fff640 10 monclient(hunting): picked mon.noname-c con 0x7fead80b38c0 addr v1:172.21.15.135:6790/0 2024-03-29T11:34:20.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.465+0000 7fead7fff640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fead80aed90 con 0x7fead80b2800 2024-03-29T11:34:20.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.465+0000 7fead7fff640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fead80ad040 con 0x7fead80a9c00 2024-03-29T11:34:20.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.465+0000 7fead7fff640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fead813e530 con 0x7fead80b38c0 2024-03-29T11:34:20.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.465+0000 7fead7fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:20.464 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.465+0000 7fead7fff640 10 monclient(hunting): authenticate will time out at 1367.834161s 2024-03-29T11:34:20.466 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.468+0000 7fead77fe640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7fead80b38c0 0x7fead80b04f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:45471/0 (socket says 172.21.15.135:45471) 2024-03-29T11:34:20.467 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.468+0000 7fead77fe640 1 -- 172.21.15.135:0/1912973417 learned_addr learned my addr 172.21.15.135:0/1912973417 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:34:20.467 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.468+0000 7feacffff640 10 client.?.objecter ms_handle_connect 0x7fead80b38c0 2024-03-29T11:34:20.467 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.468+0000 7feacffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:20.467 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.469+0000 7feacffff640 10 client.?.objecter ms_handle_connect 0x7fead80b2800 2024-03-29T11:34:20.467 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.469+0000 7feacffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:20.470 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.469+0000 7feacffff640 10 client.?.objecter ms_handle_connect 0x7fead80a9c00 2024-03-29T11:34:20.470 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.469+0000 7feacffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:20.471 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.469+0000 7feacffff640 1 -- 172.21.15.135:0/1912973417 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368200757 0 0) 0x7fead813e530 con 0x7fead80b38c0 2024-03-29T11:34:20.471 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.469+0000 7feacffff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:20.471 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.469+0000 7feacffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:20.471 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.469+0000 7feacffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:20.471 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.469+0000 7feacffff640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:34:20.471 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.469+0000 7feacffff640 1 -- 172.21.15.135:0/1912973417 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feab8004eb0 con 0x7fead80b38c0 2024-03-29T11:34:20.471 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.469+0000 7feacffff640 1 -- 172.21.15.135:0/1912973417 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994800634 0 0) 0x7fead80aed90 con 0x7fead80b2800 2024-03-29T11:34:20.471 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.469+0000 7feacffff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:20.471 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.469+0000 7feacffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:20.471 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.469+0000 7feacffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:20.471 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.469+0000 7feacffff640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:34:20.471 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.469+0000 7feacffff640 1 -- 172.21.15.135:0/1912973417 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fead813e530 con 0x7fead80b2800 2024-03-29T11:34:20.471 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.470+0000 7feacffff640 1 -- 172.21.15.135:0/1912973417 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4294104815 0 0) 0x7fead80ad040 con 0x7fead80a9c00 2024-03-29T11:34:20.472 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.470+0000 7feacffff640 10 monclient(hunting): _negotiate 2024-03-29T11:34:20.472 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.470+0000 7feacffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:20.472 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.470+0000 7feacffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:20.472 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.470+0000 7feacffff640 10 monclient(hunting): my global_id is 4576 2024-03-29T11:34:20.472 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.470+0000 7feacffff640 1 -- 172.21.15.135:0/1912973417 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fead80aed90 con 0x7fead80a9c00 2024-03-29T11:34:20.472 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.470+0000 7feacffff640 1 -- 172.21.15.135:0/1912973417 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4060254213 0 0) 0x7fead813e530 con 0x7fead80b2800 2024-03-29T11:34:20.472 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.470+0000 7feacffff640 1 -- 172.21.15.135:0/1912973417 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fead80ad040 con 0x7fead80b2800 2024-03-29T11:34:20.472 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.470+0000 7feacffff640 1 -- 172.21.15.135:0/1912973417 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7feac80017f0 con 0x7fead80b2800 2024-03-29T11:34:20.472 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.470+0000 7feacffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:20.472 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.470+0000 7feacffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:20.472 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.470+0000 7feacffff640 10 monclient(hunting): dump: 2024-03-29T11:34:20.472 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:20.472 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:20.472 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:20.473 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:20.473 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:20.473 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:20.473 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:20.473 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:20.473 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:20.473 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:20.473 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.470+0000 7feacffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:20.473 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.470+0000 7feacffff640 1 -- 172.21.15.135:0/1912973417 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3805365789 0 0) 0x7fead80aed90 con 0x7fead80a9c00 2024-03-29T11:34:20.473 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.470+0000 7feacffff640 1 -- 172.21.15.135:0/1912973417 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fead813e530 con 0x7fead80a9c00 2024-03-29T11:34:20.473 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.470+0000 7feacffff640 1 -- 172.21.15.135:0/1912973417 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2528950204 0 0) 0x7fead80ad040 con 0x7fead80b2800 2024-03-29T11:34:20.473 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.470+0000 7feacffff640 1 -- 172.21.15.135:0/1912973417 >> v1:172.21.15.135:6790/0 conn(0x7fead80b38c0 legacy=0x7fead80b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:20.473 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.470+0000 7feacffff640 1 -- 172.21.15.135:0/1912973417 >> v1:172.21.15.194:6789/0 conn(0x7fead80a9c00 legacy=0x7fead80ac6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:20.473 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.470+0000 7feacffff640 10 monclient: _finish_hunting 0 2024-03-29T11:34:20.474 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.470+0000 7feacffff640 1 monclient: found mon.a 2024-03-29T11:34:20.474 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.470+0000 7feacffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:20.474 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.470+0000 7feacffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:34:20.474 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.470+0000 7feacffff640 1 -- 172.21.15.135:0/1912973417 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead813e6c0 con 0x7fead80b2800 2024-03-29T11:34:20.474 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.470+0000 7feacffff640 10 monclient: _finish_auth 0 2024-03-29T11:34:20.474 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.470+0000 7feacffff640 10 monclient: _check_auth_tickets 2024-03-29T11:34:20.474 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.470+0000 7feacffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:20.474 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.471+0000 7feacffff640 1 -- 172.21.15.135:0/1912973417 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feac80019d0 con 0x7fead80b2800 2024-03-29T11:34:20.474 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.471+0000 7feacffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:20.474 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.471+0000 7feacffff640 1 -- 172.21.15.135:0/1912973417 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7feac80037d0 con 0x7fead80b2800 2024-03-29T11:34:20.474 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.471+0000 7feacffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:20.474 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.471+0000 7feacffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:20.474 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.471+0000 7feacffff640 10 monclient: dump: 2024-03-29T11:34:20.474 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:20.475 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:20.475 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:20.475 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:20.475 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:20.475 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:20.475 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:20.475 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:20.475 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:20.475 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:20.475 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.471+0000 7fead7fff640 5 monclient: authenticate success, global_id 4554 2024-03-29T11:34:20.475 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.471+0000 7fead7fff640 10 monclient: _renew_subs 2024-03-29T11:34:20.475 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.471+0000 7fead7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:34:20.475 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.471+0000 7fead7fff640 1 -- 172.21.15.135:0/1912973417 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fead80b0c00 con 0x7fead80b2800 2024-03-29T11:34:20.475 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.471+0000 7fead7fff640 10 client.4554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:20.476 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.471+0000 7fead7fff640 10 monclient: _renew_subs 2024-03-29T11:34:20.476 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.471+0000 7fead7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:34:20.476 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.471+0000 7fead7fff640 1 -- 172.21.15.135:0/1912973417 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fead81d6cc0 con 0x7fead80b2800 2024-03-29T11:34:20.476 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.471+0000 7fead7fff640 1 librados: init done 2024-03-29T11:34:20.476 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.472+0000 7fead5ffb640 4 set_mon_vals no callback set 2024-03-29T11:34:20.477 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.473+0000 7fead7fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:20.477 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.473+0000 7feacffff640 1 -- 172.21.15.135:0/1912973417 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7feac80197a0 con 0x7fead80b2800 2024-03-29T11:34:20.477 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.474+0000 7fead7fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:20.478 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.475+0000 7fead7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:34:20.478 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.475+0000 7fead7fff640 1 -- 172.21.15.135:0/1912973417 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feabc005080 con 0x7fead80b2800 2024-03-29T11:34:20.478 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.475+0000 7feacffff640 1 -- 172.21.15.135:0/1912973417 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3808419264 0 0) 0x7feac8084820 con 0x7fead80b2800 2024-03-29T11:34:20.478 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.475+0000 7feacffff640 10 client.4554.objecter ms_dispatch 0x7fead8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:34:20.478 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.476+0000 7fead6ffd640 10 monclient: get_auth_request con 0x7feab8076820 auth_method 0 2024-03-29T11:34:20.478 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.476+0000 7feacffff640 3 client.4554.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:34:20.478 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.476+0000 7feacffff640 3 client.4554.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:34:20.478 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.476+0000 7feacffff640 20 client.4554.objecter dump_active .. 0 homeless 2024-03-29T11:34:20.478 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.477+0000 7feacffff640 10 client.4554.objecter ms_handle_connect 0x7feab8076820 2024-03-29T11:34:20.483 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.485+0000 7feacffff640 1 -- 172.21.15.135:0/1912973417 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feac804ef00 con 0x7fead80b2800 2024-03-29T11:34:20.483 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.486+0000 7feacffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:20.483 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.486+0000 7feacffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:20.811 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.813+0000 7fead7fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:20.811 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.813+0000 7fead7fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:20.811 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.813+0000 7fead7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:34:20.811 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.813+0000 7fead7fff640 1 -- 172.21.15.135:0/1912973417 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feabc005330 con 0x7fead80b2800 2024-03-29T11:34:20.812 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.814+0000 7feacffff640 1 -- 172.21.15.135:0/1912973417 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1242783157) 0x7feac8051000 con 0x7fead80b2800 2024-03-29T11:34:20.812 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.814+0000 7feacffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:20.812 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.814+0000 7feacffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:20.812 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:34:20.813 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":10,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:24:24.156239+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:24.156239+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:20.816 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.818+0000 7fead7fff640 10 librados: watch_flush enter 2024-03-29T11:34:20.817 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.818+0000 7fead7fff640 10 librados: watch_flush exit 2024-03-29T11:34:20.817 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.818+0000 7fead7fff640 20 client.4554.objecter shutdown clearing up homeless session... 2024-03-29T11:34:20.817 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.818+0000 7fead7fff640 10 client.4554.objecter successfully canceled tick 2024-03-29T11:34:20.818 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.819+0000 7fead7fff640 1 -- 172.21.15.135:0/1912973417 >> v1:172.21.15.135:6816/3469875285 conn(0x7feab8076820 legacy=0x7feab8078ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:20.818 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.819+0000 7fead7fff640 10 monclient: shutdown 2024-03-29T11:34:20.818 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.819+0000 7fead7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:20.818 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.819+0000 7fead7fff640 1 -- 172.21.15.135:0/1912973417 >> v1:172.21.15.135:6789/0 conn(0x7fead80b2800 legacy=0x7fead80ad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:20.818 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.819+0000 7fead7fff640 1 -- 172.21.15.135:0/1912973417 shutdown_connections 2024-03-29T11:34:20.818 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.819+0000 7fead7fff640 1 -- 172.21.15.135:0/1912973417 >> 172.21.15.135:0/1912973417 conn(0x7fead8066a00 msgr2=0x7fead8140470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:20.818 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.819+0000 7fead7fff640 1 -- 172.21.15.135:0/1912973417 shutdown_connections 2024-03-29T11:34:20.818 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.819+0000 7fead7fff640 1 -- 172.21.15.135:0/1912973417 wait complete. 2024-03-29T11:34:20.818 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:20.820+0000 7fead7fff640 1 librados: shutdown 2024-03-29T11:34:20.835 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:27.641 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:27.795 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.797+0000 7fc4a7fff640 10 monclient: get_monmap_and_config 2024-03-29T11:34:27.795 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.797+0000 7fc4a7fff640 10 monclient: build_initial_monmap 2024-03-29T11:34:27.795 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.797+0000 7fc4a7fff640 10 monclient: monmap: 2024-03-29T11:34:27.795 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:34:27.795 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:27.796 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:34:27.798810+0000 2024-03-29T11:34:27.796 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:34:27.798810+0000 2024-03-29T11:34:27.796 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:27.796 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:27.796 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:34:27.796 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:34:27.796 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:34:27.796 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:27.797 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.798+0000 7fc4a7fff640 1 Processor -- start 2024-03-29T11:34:27.797 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.798+0000 7fc4a7fff640 1 -- start start 2024-03-29T11:34:27.797 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.798+0000 7fc4a7fff640 10 monclient: init 2024-03-29T11:34:27.798 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.799+0000 7fc4a7fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:27.798 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.799+0000 7fc4a7fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:34:27.798 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.799+0000 7fc4a7fff640 10 monclient(hunting): picked mon.noname-c con 0x7fc4a80b2800 addr v1:172.21.15.194:6789/0 2024-03-29T11:34:27.798 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.799+0000 7fc4a7fff640 10 monclient(hunting): picked mon.noname-a con 0x7fc4a80a9c00 addr v1:172.21.15.135:6789/0 2024-03-29T11:34:27.798 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.799+0000 7fc4a7fff640 10 monclient(hunting): picked mon.noname-b con 0x7fc4a80b38c0 addr v1:172.21.15.135:6790/0 2024-03-29T11:34:27.798 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.799+0000 7fc4a7fff640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a8141eb0 con 0x7fc4a80a9c00 2024-03-29T11:34:27.798 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.799+0000 7fc4a7fff640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a80b22d0 con 0x7fc4a80b2800 2024-03-29T11:34:27.798 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.799+0000 7fc4a7fff640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a8142d30 con 0x7fc4a80b38c0 2024-03-29T11:34:27.798 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.799+0000 7fc4a7fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:27.798 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.799+0000 7fc4a7fff640 10 monclient(hunting): authenticate will time out at 1375.168270s 2024-03-29T11:34:27.799 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.802+0000 7fc4a6ffd640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fc4a80b2800 0x7fc4a8142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:37567/0 (socket says 172.21.15.135:37567) 2024-03-29T11:34:27.799 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.802+0000 7fc4a6ffd640 1 -- 172.21.15.135:0/371939548 learned_addr learned my addr 172.21.15.135:0/371939548 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:34:27.801 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.802+0000 7fc4a5ffb640 1 -- 172.21.15.135:0/371939548 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2772750763 0 0) 0x7fc4a80b22d0 con 0x7fc4a80b2800 2024-03-29T11:34:27.801 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.802+0000 7fc4a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:27.801 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.802+0000 7fc4a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:27.802 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.802+0000 7fc4a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:27.802 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.802+0000 7fc4a5ffb640 10 monclient(hunting): my global_id is 4579 2024-03-29T11:34:27.802 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.802+0000 7fc4a5ffb640 1 -- 172.21.15.135:0/371939548 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc490003520 con 0x7fc4a80b2800 2024-03-29T11:34:27.802 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.802+0000 7fc4a5ffb640 1 -- 172.21.15.135:0/371939548 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265057861 0 0) 0x7fc4a8141eb0 con 0x7fc4a80a9c00 2024-03-29T11:34:27.802 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.802+0000 7fc4a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:27.802 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.802+0000 7fc4a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:27.802 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.802+0000 7fc4a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:27.802 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.802+0000 7fc4a5ffb640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:34:27.802 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.802+0000 7fc4a5ffb640 1 -- 172.21.15.135:0/371939548 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4a80b22d0 con 0x7fc4a80a9c00 2024-03-29T11:34:27.802 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 1 -- 172.21.15.135:0/371939548 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 543513471 0 0) 0x7fc4a8142d30 con 0x7fc4a80b38c0 2024-03-29T11:34:27.802 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:34:27.803 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:27.803 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:27.803 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:34:27.803 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 1 -- 172.21.15.135:0/371939548 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4a8141eb0 con 0x7fc4a80b38c0 2024-03-29T11:34:27.803 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 1 -- 172.21.15.135:0/371939548 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 784599590 0 0) 0x7fc4a80b22d0 con 0x7fc4a80a9c00 2024-03-29T11:34:27.803 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 1 -- 172.21.15.135:0/371939548 >> v1:172.21.15.135:6790/0 conn(0x7fc4a80b38c0 legacy=0x7fc4a814d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:27.803 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 1 -- 172.21.15.135:0/371939548 >> v1:172.21.15.194:6789/0 conn(0x7fc4a80b2800 legacy=0x7fc4a8142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:27.803 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:34:27.803 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 1 monclient: found mon.noname-a 2024-03-29T11:34:27.803 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:27.803 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.135:6789/0 2024-03-29T11:34:27.803 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 1 -- 172.21.15.135:0/371939548 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a80aaf90 con 0x7fc4a80a9c00 2024-03-29T11:34:27.803 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:34:27.803 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:34:27.804 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:27.804 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 1 -- 172.21.15.135:0/371939548 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fc4980015f0 con 0x7fc4a80a9c00 2024-03-29T11:34:27.804 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:27.804 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:27.804 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 10 monclient: dump: 2024-03-29T11:34:27.804 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:27.804 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:27.804 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:27.804 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:27.804 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:27.804 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:27.804 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:27.805 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:27.805 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:27.805 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:27.805 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 1 -- 172.21.15.135:0/371939548 <== mon.0 v1:172.21.15.135:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc498001810 con 0x7fc4a80a9c00 2024-03-29T11:34:27.805 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:27.805 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 1 -- 172.21.15.135:0/371939548 <== mon.0 v1:172.21.15.135:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fc498001ba0 con 0x7fc4a80a9c00 2024-03-29T11:34:27.805 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:27.805 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:27.805 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.803+0000 7fc4a5ffb640 10 monclient: dump: 2024-03-29T11:34:27.805 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:27.806 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:27.806 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:27.806 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:27.806 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:27.806 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:27.806 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:27.806 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:27.806 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:27.806 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:27.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.804+0000 7fc4a7fff640 5 monclient: authenticate success, global_id 4557 2024-03-29T11:34:27.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.804+0000 7fc4a7fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:27.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.804+0000 7fc4a7fff640 4 set_mon_vals no callback set 2024-03-29T11:34:27.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.805+0000 7fc4a7fff640 10 monclient: shutdown 2024-03-29T11:34:27.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.805+0000 7fc4a7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:27.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.805+0000 7fc4a7fff640 1 -- 172.21.15.135:0/371939548 >> v1:172.21.15.135:6789/0 conn(0x7fc4a80a9c00 legacy=0x7fc4a813bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:27.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.805+0000 7fc4a7fff640 1 -- 172.21.15.135:0/371939548 shutdown_connections 2024-03-29T11:34:27.810 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.805+0000 7fc4a7fff640 1 -- 172.21.15.135:0/371939548 >> 172.21.15.135:0/371939548 conn(0x7fc4a8066a00 msgr2=0x7fc4a8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:27.811 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.807+0000 7fc4a7fff640 1 -- 172.21.15.135:0/371939548 shutdown_connections 2024-03-29T11:34:27.811 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.807+0000 7fc4a7fff640 1 -- 172.21.15.135:0/371939548 wait complete. 2024-03-29T11:34:27.811 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.808+0000 7fc4a7fff640 10 monclient: build_initial_monmap 2024-03-29T11:34:27.811 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.808+0000 7fc4a7fff640 10 monclient: monmap: 2024-03-29T11:34:27.811 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:34:27.811 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:27.811 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:34:27.811 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:34:27.811 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:27.811 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:27.812 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:34:27.812 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:34:27.812 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:34:27.812 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:27.812 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.809+0000 7fc4a7fff640 1 librados: starting msgr at 2024-03-29T11:34:27.812 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.809+0000 7fc4a7fff640 1 librados: starting objecter 2024-03-29T11:34:27.812 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.809+0000 7fc4a7fff640 1 Processor -- start 2024-03-29T11:34:27.812 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.809+0000 7fc4a7fff640 1 -- start start 2024-03-29T11:34:27.812 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.809+0000 7fc4a7fff640 1 librados: setting wanted keys 2024-03-29T11:34:27.813 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.809+0000 7fc4a7fff640 1 librados: calling monclient init 2024-03-29T11:34:27.813 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.809+0000 7fc4a7fff640 10 monclient: init 2024-03-29T11:34:27.813 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.810+0000 7fc4a7fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:27.813 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.810+0000 7fc4a7fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:27.813 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.810+0000 7fc4a7fff640 10 monclient(hunting): picked mon.noname-b con 0x7fc4a80b2800 addr v1:172.21.15.194:6789/0 2024-03-29T11:34:27.813 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.810+0000 7fc4a7fff640 10 monclient(hunting): picked mon.noname-c con 0x7fc4a80b38c0 addr v1:172.21.15.135:6790/0 2024-03-29T11:34:27.813 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.810+0000 7fc4a7fff640 10 monclient(hunting): picked mon.noname-a con 0x7fc4a813cf50 addr v1:172.21.15.135:6789/0 2024-03-29T11:34:27.813 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.810+0000 7fc4a7fff640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a806f990 con 0x7fc4a813cf50 2024-03-29T11:34:27.813 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.810+0000 7fc4a7fff640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a806fc70 con 0x7fc4a80b2800 2024-03-29T11:34:27.813 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.810+0000 7fc4a7fff640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a813d3e0 con 0x7fc4a80b38c0 2024-03-29T11:34:27.813 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.810+0000 7fc4a7fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:27.813 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.810+0000 7fc4a7fff640 10 monclient(hunting): authenticate will time out at 1375.179556s 2024-03-29T11:34:27.815 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.813+0000 7fc4a77fe640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fc4a813cf50 0x7fc4a81e63f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:48763/0 (socket says 172.21.15.135:48763) 2024-03-29T11:34:27.816 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.813+0000 7fc4a77fe640 1 -- 172.21.15.135:0/961131665 learned_addr learned my addr 172.21.15.135:0/961131665 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:34:27.816 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.816+0000 7fc48f7fe640 10 client.?.objecter ms_handle_connect 0x7fc4a813cf50 2024-03-29T11:34:27.816 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.816+0000 7fc48f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:27.816 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.816+0000 7fc48f7fe640 10 client.?.objecter ms_handle_connect 0x7fc4a80b38c0 2024-03-29T11:34:27.816 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.816+0000 7fc48f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:27.816 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.816+0000 7fc48f7fe640 10 client.?.objecter ms_handle_connect 0x7fc4a80b2800 2024-03-29T11:34:27.816 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.816+0000 7fc48f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:27.817 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.817+0000 7fc48f7fe640 1 -- 172.21.15.135:0/961131665 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3548466224 0 0) 0x7fc4a806f990 con 0x7fc4a813cf50 2024-03-29T11:34:27.817 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.817+0000 7fc48f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:27.817 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.817+0000 7fc48f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:27.817 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.817+0000 7fc48f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:27.817 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.817+0000 7fc48f7fe640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:34:27.817 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.817+0000 7fc48f7fe640 1 -- 172.21.15.135:0/961131665 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc494004eb0 con 0x7fc4a813cf50 2024-03-29T11:34:27.817 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.817+0000 7fc48f7fe640 1 -- 172.21.15.135:0/961131665 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 327433178 0 0) 0x7fc4a813d3e0 con 0x7fc4a80b38c0 2024-03-29T11:34:27.817 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.817+0000 7fc48f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:27.817 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.817+0000 7fc48f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:27.818 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.817+0000 7fc48f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:27.818 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.817+0000 7fc48f7fe640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:34:27.818 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.817+0000 7fc48f7fe640 1 -- 172.21.15.135:0/961131665 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4a806f990 con 0x7fc4a80b38c0 2024-03-29T11:34:27.818 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.817+0000 7fc48f7fe640 1 -- 172.21.15.135:0/961131665 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3051034067 0 0) 0x7fc494004eb0 con 0x7fc4a813cf50 2024-03-29T11:34:27.818 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.817+0000 7fc48f7fe640 1 -- 172.21.15.135:0/961131665 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4940064f0 con 0x7fc4a813cf50 2024-03-29T11:34:27.818 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.817+0000 7fc48f7fe640 1 -- 172.21.15.135:0/961131665 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982416632 0 0) 0x7fc4a806fc70 con 0x7fc4a80b2800 2024-03-29T11:34:27.818 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.817+0000 7fc48f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:27.818 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.817+0000 7fc48f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:27.819 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.817+0000 7fc48f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:27.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.817+0000 7fc48f7fe640 10 monclient(hunting): my global_id is 4582 2024-03-29T11:34:27.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.817+0000 7fc48f7fe640 1 -- 172.21.15.135:0/961131665 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc494006930 con 0x7fc4a80b2800 2024-03-29T11:34:27.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.817+0000 7fc48f7fe640 1 -- 172.21.15.135:0/961131665 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2659690134 0 0) 0x7fc4a806f990 con 0x7fc4a80b38c0 2024-03-29T11:34:27.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.818+0000 7fc48f7fe640 1 -- 172.21.15.135:0/961131665 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4a806fc70 con 0x7fc4a80b38c0 2024-03-29T11:34:27.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.818+0000 7fc48f7fe640 1 -- 172.21.15.135:0/961131665 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fc498001530 con 0x7fc4a80b38c0 2024-03-29T11:34:27.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.818+0000 7fc48f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:27.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.818+0000 7fc48f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:27.824 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.818+0000 7fc48f7fe640 10 monclient(hunting): dump: 2024-03-29T11:34:27.824 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:27.824 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:27.824 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:27.825 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:27.825 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:27.825 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:27.825 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:27.825 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:27.825 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:27.825 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:27.825 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.818+0000 7fc48f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:27.825 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.819+0000 7fc48f7fe640 1 -- 172.21.15.135:0/961131665 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1468228929 0 0) 0x7fc4a806fc70 con 0x7fc4a80b38c0 2024-03-29T11:34:27.825 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.819+0000 7fc48f7fe640 1 -- 172.21.15.135:0/961131665 >> v1:172.21.15.194:6789/0 conn(0x7fc4a80b2800 legacy=0x7fc4a806de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:27.825 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.819+0000 7fc48f7fe640 1 -- 172.21.15.135:0/961131665 >> v1:172.21.15.135:6789/0 conn(0x7fc4a813cf50 legacy=0x7fc4a81e63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:27.825 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.819+0000 7fc48f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:34:27.825 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.819+0000 7fc48f7fe640 1 monclient: found mon.c 2024-03-29T11:34:27.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.819+0000 7fc48f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:27.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.819+0000 7fc48f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:34:27.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.819+0000 7fc48f7fe640 1 -- 172.21.15.135:0/961131665 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a8071f60 con 0x7fc4a80b38c0 2024-03-29T11:34:27.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.819+0000 7fc48f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:34:27.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.819+0000 7fc48f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:34:27.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.819+0000 7fc48f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:27.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.820+0000 7fc4a7fff640 5 monclient: authenticate success, global_id 4607 2024-03-29T11:34:27.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.820+0000 7fc4a7fff640 10 monclient: _renew_subs 2024-03-29T11:34:27.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.820+0000 7fc4a7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:34:27.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.820+0000 7fc4a7fff640 1 -- 172.21.15.135:0/961131665 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a806e5a0 con 0x7fc4a80b38c0 2024-03-29T11:34:27.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.820+0000 7fc4a7fff640 10 client.4607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:27.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.820+0000 7fc4a7fff640 10 monclient: _renew_subs 2024-03-29T11:34:27.826 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.820+0000 7fc4a7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:34:27.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.820+0000 7fc4a7fff640 1 -- 172.21.15.135:0/961131665 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a81e7cf0 con 0x7fc4a80b38c0 2024-03-29T11:34:27.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.820+0000 7fc4a7fff640 1 librados: init done 2024-03-29T11:34:27.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.822+0000 7fc48f7fe640 1 -- 172.21.15.135:0/961131665 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc498001690 con 0x7fc4a80b38c0 2024-03-29T11:34:27.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.822+0000 7fc48f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:27.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.822+0000 7fc48f7fe640 1 -- 172.21.15.135:0/961131665 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fc498001210 con 0x7fc4a80b38c0 2024-03-29T11:34:27.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.822+0000 7fc48f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:27.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.822+0000 7fc48f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:34:27.827 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.822+0000 7fc48f7fe640 10 monclient: dump: 2024-03-29T11:34:27.827 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:27.827 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:27.827 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:27.827 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:27.827 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:27.827 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:27.828 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:27.828 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:27.828 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:27.828 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:27.828 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.822+0000 7fc4a7fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:27.828 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.822+0000 7fc4a7fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:27.828 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.822+0000 7fc4a7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:34:27.828 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.822+0000 7fc4a7fff640 1 -- 172.21.15.135:0/961131665 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4a8070190 con 0x7fc4a80b38c0 2024-03-29T11:34:27.828 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.823+0000 7fc48f7fe640 1 -- 172.21.15.135:0/961131665 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fc498019930 con 0x7fc4a80b38c0 2024-03-29T11:34:27.828 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.822+0000 7fc4a4ff9640 4 set_mon_vals no callback set 2024-03-29T11:34:27.828 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.824+0000 7fc48f7fe640 1 -- 172.21.15.135:0/961131665 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1288947923 0 0) 0x7fc4980533a0 con 0x7fc4a80b38c0 2024-03-29T11:34:27.828 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.824+0000 7fc48f7fe640 10 client.4607.objecter ms_dispatch 0x7fc4a8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:34:27.828 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.824+0000 7fc48f7fe640 3 client.4607.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:34:27.828 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.824+0000 7fc48f7fe640 3 client.4607.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:34:27.829 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.824+0000 7fc48f7fe640 20 client.4607.objecter dump_active .. 0 homeless 2024-03-29T11:34:27.829 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.826+0000 7fc4a6ffd640 10 monclient: get_auth_request con 0x7fc49406e1b0 auth_method 0 2024-03-29T11:34:27.829 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.827+0000 7fc48f7fe640 10 client.4607.objecter ms_handle_connect 0x7fc49406e1b0 2024-03-29T11:34:27.830 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.832+0000 7fc48f7fe640 1 -- 172.21.15.135:0/961131665 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc49804f640 con 0x7fc4a80b38c0 2024-03-29T11:34:27.831 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.832+0000 7fc48f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:27.831 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:27.832+0000 7fc48f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:28.156 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:28.158+0000 7fc4a7fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:28.156 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:28.158+0000 7fc4a7fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:28.156 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:28.158+0000 7fc4a7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:34:28.156 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:28.158+0000 7fc4a7fff640 1 -- 172.21.15.135:0/961131665 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4a8070190 con 0x7fc4a80b38c0 2024-03-29T11:34:28.159 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:28.161+0000 7fc48f7fe640 1 -- 172.21.15.135:0/961131665 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2351702130) 0x7fc4a8070190 con 0x7fc4a80b38c0 2024-03-29T11:34:28.159 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:28.161+0000 7fc48f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:28.159 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:28.161+0000 7fc48f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:28.159 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:34:28.159 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":11,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:34:26.903077+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:26.903077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:28.163 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:28.165+0000 7fc4a7fff640 10 librados: watch_flush enter 2024-03-29T11:34:28.163 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:28.166+0000 7fc4a7fff640 10 librados: watch_flush exit 2024-03-29T11:34:28.163 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:28.166+0000 7fc4a7fff640 20 client.4607.objecter shutdown clearing up homeless session... 2024-03-29T11:34:28.164 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:28.166+0000 7fc4a7fff640 10 client.4607.objecter successfully canceled tick 2024-03-29T11:34:28.164 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:28.166+0000 7fc4a7fff640 1 -- 172.21.15.135:0/961131665 >> v1:172.21.15.135:6816/3469875285 conn(0x7fc49406e1b0 legacy=0x7fc494070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:28.164 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:28.166+0000 7fc4a7fff640 10 monclient: shutdown 2024-03-29T11:34:28.164 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:28.166+0000 7fc4a7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:28.164 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:28.166+0000 7fc4a7fff640 1 -- 172.21.15.135:0/961131665 >> v1:172.21.15.135:6790/0 conn(0x7fc4a80b38c0 legacy=0x7fc4a813c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:28.165 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:28.168+0000 7fc4a7fff640 1 -- 172.21.15.135:0/961131665 shutdown_connections 2024-03-29T11:34:28.166 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:28.168+0000 7fc4a7fff640 1 -- 172.21.15.135:0/961131665 >> 172.21.15.135:0/961131665 conn(0x7fc4a8066a00 msgr2=0x7fc4a81404e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:28.166 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:28.168+0000 7fc4a7fff640 1 -- 172.21.15.135:0/961131665 shutdown_connections 2024-03-29T11:34:28.166 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:28.168+0000 7fc4a7fff640 1 -- 172.21.15.135:0/961131665 wait complete. 2024-03-29T11:34:28.166 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:28.168+0000 7fc4a7fff640 1 librados: shutdown 2024-03-29T11:34:28.183 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:34.991 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:35.146 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.147+0000 7f8f408b3640 10 monclient: get_monmap_and_config 2024-03-29T11:34:35.146 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.147+0000 7f8f408b3640 10 monclient: build_initial_monmap 2024-03-29T11:34:35.146 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.147+0000 7f8f408b3640 10 monclient: monmap: 2024-03-29T11:34:35.146 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:34:35.146 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:35.146 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:34:35.149332+0000 2024-03-29T11:34:35.146 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:34:35.149332+0000 2024-03-29T11:34:35.146 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:35.146 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:35.146 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:34:35.147 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:34:35.147 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:34:35.147 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:35.147 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.148+0000 7f8f408b3640 1 Processor -- start 2024-03-29T11:34:35.147 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.148+0000 7f8f408b3640 1 -- start start 2024-03-29T11:34:35.147 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.148+0000 7f8f408b3640 10 monclient: init 2024-03-29T11:34:35.150 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.149+0000 7f8f408b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:35.150 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.149+0000 7f8f408b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:34:35.150 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.149+0000 7f8f408b3640 10 monclient(hunting): picked mon.noname-c con 0x7f8f3c0b0770 addr v1:172.21.15.194:6789/0 2024-03-29T11:34:35.150 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.149+0000 7f8f408b3640 10 monclient(hunting): picked mon.noname-b con 0x7f8f3c14a700 addr v1:172.21.15.135:6790/0 2024-03-29T11:34:35.150 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.149+0000 7f8f408b3640 10 monclient(hunting): picked mon.noname-a con 0x7f8f3c0ac5a0 addr v1:172.21.15.135:6789/0 2024-03-29T11:34:35.150 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.149+0000 7f8f408b3640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f3c0aba70 con 0x7f8f3c0ac5a0 2024-03-29T11:34:35.150 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.149+0000 7f8f408b3640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f3c0aecb0 con 0x7f8f3c0b0770 2024-03-29T11:34:35.150 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.149+0000 7f8f408b3640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f3c0af150 con 0x7f8f3c14a700 2024-03-29T11:34:35.150 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.149+0000 7f8f408b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:35.150 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.149+0000 7f8f408b3640 10 monclient(hunting): authenticate will time out at 1382.518891s 2024-03-29T11:34:35.151 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.153+0000 7f8f3ad9d640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f8f3c0ac5a0 0x7f8f3c151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:41591/0 (socket says 172.21.15.135:41591) 2024-03-29T11:34:35.151 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.153+0000 7f8f3ad9d640 1 -- 172.21.15.135:0/3954380845 learned_addr learned my addr 172.21.15.135:0/3954380845 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:34:35.152 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.154+0000 7f8f3959a640 1 -- 172.21.15.135:0/3954380845 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 790163883 0 0) 0x7f8f3c0aba70 con 0x7f8f3c0ac5a0 2024-03-29T11:34:35.152 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.154+0000 7f8f3959a640 10 monclient(hunting): _negotiate 2024-03-29T11:34:35.152 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.154+0000 7f8f3959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:35.152 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.154+0000 7f8f3959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:35.152 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.154+0000 7f8f3959a640 10 monclient(hunting): my global_id is 4563 2024-03-29T11:34:35.152 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.154+0000 7f8f3959a640 1 -- 172.21.15.135:0/3954380845 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f24003520 con 0x7f8f3c0ac5a0 2024-03-29T11:34:35.157 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.157+0000 7f8f3959a640 1 -- 172.21.15.135:0/3954380845 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1192508842 0 0) 0x7f8f24003520 con 0x7f8f3c0ac5a0 2024-03-29T11:34:35.157 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.157+0000 7f8f3959a640 1 -- 172.21.15.135:0/3954380845 >> v1:172.21.15.135:6790/0 conn(0x7f8f3c14a700 legacy=0x7f8f3c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:35.157 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.157+0000 7f8f3959a640 1 -- 172.21.15.135:0/3954380845 >> v1:172.21.15.194:6789/0 conn(0x7f8f3c0b0770 legacy=0x7f8f3c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:35.157 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.157+0000 7f8f3959a640 10 monclient: _finish_hunting 0 2024-03-29T11:34:35.157 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.157+0000 7f8f3959a640 1 monclient: found mon.noname-a 2024-03-29T11:34:35.157 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.158+0000 7f8f3959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:35.158 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.158+0000 7f8f3959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.135:6789/0 2024-03-29T11:34:35.158 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.158+0000 7f8f3959a640 1 -- 172.21.15.135:0/3954380845 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3c0ac9d0 con 0x7f8f3c0ac5a0 2024-03-29T11:34:35.158 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.158+0000 7f8f3959a640 10 monclient: _finish_auth 0 2024-03-29T11:34:35.158 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.158+0000 7f8f3959a640 10 monclient: _check_auth_tickets 2024-03-29T11:34:35.158 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.158+0000 7f8f3959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:35.158 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.158+0000 7f8f408b3640 5 monclient: authenticate success, global_id 4563 2024-03-29T11:34:35.158 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.158+0000 7f8f408b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:35.158 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.158+0000 7f8f3959a640 1 -- 172.21.15.135:0/3954380845 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f8f34001a10 con 0x7f8f3c0ac5a0 2024-03-29T11:34:35.158 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.158+0000 7f8f3959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:35.159 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.158+0000 7f8f3959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:35.159 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.158+0000 7f8f3959a640 10 monclient: dump: 2024-03-29T11:34:35.159 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:35.159 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:35.159 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:35.159 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:35.159 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:35.159 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:35.159 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:35.159 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:35.159 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:35.159 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:35.159 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.158+0000 7f8f408b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:35.159 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.158+0000 7f8f3959a640 1 -- 172.21.15.135:0/3954380845 <== mon.0 v1:172.21.15.135:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f34001c30 con 0x7f8f3c0ac5a0 2024-03-29T11:34:35.160 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.158+0000 7f8f3959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:35.160 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.158+0000 7f8f408b3640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:35.160 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.158+0000 7f8f408b3640 4 set_mon_vals no callback set 2024-03-29T11:34:35.160 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.158+0000 7f8f3959a640 1 -- 172.21.15.135:0/3954380845 <== mon.0 v1:172.21.15.135:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f8f34001fc0 con 0x7f8f3c0ac5a0 2024-03-29T11:34:35.160 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.158+0000 7f8f3959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:35.160 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.158+0000 7f8f3959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:35.160 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.158+0000 7f8f3959a640 10 monclient: dump: 2024-03-29T11:34:35.160 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:35.160 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:35.160 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:35.160 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:35.160 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:35.160 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:35.160 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:35.160 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:35.160 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:35.161 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:35.161 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.159+0000 7f8f408b3640 10 monclient: shutdown 2024-03-29T11:34:35.161 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.159+0000 7f8f408b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:35.161 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.159+0000 7f8f408b3640 1 -- 172.21.15.135:0/3954380845 >> v1:172.21.15.135:6789/0 conn(0x7f8f3c0ac5a0 legacy=0x7f8f3c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:35.162 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.163+0000 7f8f408b3640 1 -- 172.21.15.135:0/3954380845 shutdown_connections 2024-03-29T11:34:35.163 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.163+0000 7f8f408b3640 1 -- 172.21.15.135:0/3954380845 >> 172.21.15.135:0/3954380845 conn(0x7f8f3c06b660 msgr2=0x7f8f3c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:35.163 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.166+0000 7f8f408b3640 1 -- 172.21.15.135:0/3954380845 shutdown_connections 2024-03-29T11:34:35.166 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.166+0000 7f8f408b3640 1 -- 172.21.15.135:0/3954380845 wait complete. 2024-03-29T11:34:35.166 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.167+0000 7f8f408b3640 10 monclient: build_initial_monmap 2024-03-29T11:34:35.166 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.167+0000 7f8f408b3640 10 monclient: monmap: 2024-03-29T11:34:35.166 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:34:35.166 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:35.166 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:34:35.166 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:34:35.166 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:35.167 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:35.167 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:34:35.167 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:34:35.167 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:34:35.167 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:35.167 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.168+0000 7f8f408b3640 1 librados: starting msgr at 2024-03-29T11:34:35.167 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.168+0000 7f8f408b3640 1 librados: starting objecter 2024-03-29T11:34:35.167 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.168+0000 7f8f408b3640 1 Processor -- start 2024-03-29T11:34:35.169 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.168+0000 7f8f408b3640 1 -- start start 2024-03-29T11:34:35.169 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.168+0000 7f8f408b3640 1 librados: setting wanted keys 2024-03-29T11:34:35.169 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.168+0000 7f8f408b3640 1 librados: calling monclient init 2024-03-29T11:34:35.169 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.168+0000 7f8f408b3640 10 monclient: init 2024-03-29T11:34:35.169 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.169+0000 7f8f408b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:35.169 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.169+0000 7f8f408b3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:34:35.169 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.169+0000 7f8f408b3640 10 monclient(hunting): picked mon.noname-c con 0x7f8f3c0ac5a0 addr v1:172.21.15.135:6790/0 2024-03-29T11:34:35.169 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.169+0000 7f8f408b3640 10 monclient(hunting): picked mon.noname-b con 0x7f8f3c0b0770 addr v1:172.21.15.194:6789/0 2024-03-29T11:34:35.169 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.169+0000 7f8f408b3640 10 monclient(hunting): picked mon.noname-a con 0x7f8f3c14a700 addr v1:172.21.15.135:6789/0 2024-03-29T11:34:35.169 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.169+0000 7f8f408b3640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f3c121790 con 0x7f8f3c14a700 2024-03-29T11:34:35.169 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.169+0000 7f8f408b3640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f3c120580 con 0x7f8f3c0b0770 2024-03-29T11:34:35.169 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.169+0000 7f8f408b3640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f3c121450 con 0x7f8f3c0ac5a0 2024-03-29T11:34:35.169 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.169+0000 7f8f408b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:35.169 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.169+0000 7f8f408b3640 10 monclient(hunting): authenticate will time out at 1382.538462s 2024-03-29T11:34:35.171 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.173+0000 7f8f39d9b640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f8f3c0b0770 0x7f8f3c148100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:46627/0 (socket says 172.21.15.135:46627) 2024-03-29T11:34:35.171 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.173+0000 7f8f39d9b640 1 -- 172.21.15.135:0/1958589974 learned_addr learned my addr 172.21.15.135:0/1958589974 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:34:35.171 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.173+0000 7f8f2affd640 10 client.?.objecter ms_handle_connect 0x7f8f3c0b0770 2024-03-29T11:34:35.171 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.173+0000 7f8f2affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:35.173 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.173+0000 7f8f2affd640 1 -- 172.21.15.135:0/1958589974 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1409558792 0 0) 0x7f8f3c120580 con 0x7f8f3c0b0770 2024-03-29T11:34:35.173 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.173+0000 7f8f2affd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:35.173 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.173+0000 7f8f2affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:35.173 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.173+0000 7f8f2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:35.173 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.173+0000 7f8f2affd640 10 monclient(hunting): my global_id is 4588 2024-03-29T11:34:35.173 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.173+0000 7f8f2affd640 1 -- 172.21.15.135:0/1958589974 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f1c004eb0 con 0x7f8f3c0b0770 2024-03-29T11:34:35.176 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.176+0000 7f8f2affd640 10 client.?.objecter ms_handle_connect 0x7f8f3c14a700 2024-03-29T11:34:35.176 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.176+0000 7f8f2affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:35.177 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.176+0000 7f8f2affd640 1 -- 172.21.15.135:0/1958589974 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3479837137 0 0) 0x7f8f3c121790 con 0x7f8f3c14a700 2024-03-29T11:34:35.177 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.176+0000 7f8f2affd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:35.177 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.176+0000 7f8f2affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:35.177 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.176+0000 7f8f2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:35.177 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.176+0000 7f8f2affd640 10 monclient(hunting): my global_id is 4566 2024-03-29T11:34:35.177 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.176+0000 7f8f2affd640 1 -- 172.21.15.135:0/1958589974 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f3c120580 con 0x7f8f3c14a700 2024-03-29T11:34:35.177 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.176+0000 7f8f2affd640 1 -- 172.21.15.135:0/1958589974 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1991664241 0 0) 0x7f8f1c004eb0 con 0x7f8f3c0b0770 2024-03-29T11:34:35.177 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.176+0000 7f8f2affd640 1 -- 172.21.15.135:0/1958589974 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f3c121790 con 0x7f8f3c0b0770 2024-03-29T11:34:35.177 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.176+0000 7f8f2affd640 1 -- 172.21.15.135:0/1958589974 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f8f30001990 con 0x7f8f3c0b0770 2024-03-29T11:34:35.177 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.176+0000 7f8f2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:35.177 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.176+0000 7f8f2affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:34:35.177 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.176+0000 7f8f2affd640 10 monclient(hunting): dump: 2024-03-29T11:34:35.177 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:35.177 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:35.178 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:35.178 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:35.178 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:35.178 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:35.178 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:35.178 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:35.178 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:35.178 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:35.178 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.176+0000 7f8f2affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:35.178 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.177+0000 7f8f2affd640 1 -- 172.21.15.135:0/1958589974 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 780349604 0 0) 0x7f8f3c120580 con 0x7f8f3c14a700 2024-03-29T11:34:35.178 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.177+0000 7f8f2affd640 1 -- 172.21.15.135:0/1958589974 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f1c004eb0 con 0x7f8f3c14a700 2024-03-29T11:34:35.178 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.177+0000 7f8f2affd640 1 -- 172.21.15.135:0/1958589974 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f8f34001da0 con 0x7f8f3c14a700 2024-03-29T11:34:35.178 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.177+0000 7f8f2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:35.178 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.177+0000 7f8f2affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:35.179 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.177+0000 7f8f2affd640 10 monclient(hunting): dump: 2024-03-29T11:34:35.179 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:35.179 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:35.179 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:35.179 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:35.179 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:35.179 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:35.179 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:35.179 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:35.179 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:35.179 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:35.179 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.177+0000 7f8f2affd640 1 -- 172.21.15.135:0/1958589974 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 780119696 0 0) 0x7f8f1c004eb0 con 0x7f8f3c14a700 2024-03-29T11:34:35.179 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.177+0000 7f8f2affd640 1 -- 172.21.15.135:0/1958589974 >> v1:172.21.15.135:6790/0 conn(0x7f8f3c0ac5a0 legacy=0x7f8f3c120d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:35.179 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.177+0000 7f8f2affd640 1 -- 172.21.15.135:0/1958589974 >> v1:172.21.15.194:6789/0 conn(0x7f8f3c0b0770 legacy=0x7f8f3c148100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:35.179 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.177+0000 7f8f2affd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:35.180 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.177+0000 7f8f2affd640 1 monclient: found mon.a 2024-03-29T11:34:35.180 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.177+0000 7f8f2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:35.180 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.177+0000 7f8f2affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:34:35.180 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.178+0000 7f8f2affd640 1 -- 172.21.15.135:0/1958589974 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3c148810 con 0x7f8f3c14a700 2024-03-29T11:34:35.180 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.178+0000 7f8f2affd640 10 monclient: _finish_auth 0 2024-03-29T11:34:35.180 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.178+0000 7f8f2affd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:35.180 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.178+0000 7f8f2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:35.181 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.179+0000 7f8f2affd640 1 -- 172.21.15.135:0/1958589974 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f34001fe0 con 0x7f8f3c14a700 2024-03-29T11:34:35.181 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.179+0000 7f8f2affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:35.181 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.179+0000 7f8f2affd640 1 -- 172.21.15.135:0/1958589974 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f8f34005110 con 0x7f8f3c14a700 2024-03-29T11:34:35.181 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.179+0000 7f8f2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:35.182 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.179+0000 7f8f2affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:35.182 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.179+0000 7f8f2affd640 10 monclient: dump: 2024-03-29T11:34:35.182 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:35.182 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:35.182 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:35.182 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:35.182 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:35.182 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:35.182 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:35.182 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:35.182 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:35.182 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:35.182 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.179+0000 7f8f38d99640 4 set_mon_vals no callback set 2024-03-29T11:34:35.182 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.180+0000 7f8f408b3640 5 monclient: authenticate success, global_id 4566 2024-03-29T11:34:35.183 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.180+0000 7f8f408b3640 10 monclient: _renew_subs 2024-03-29T11:34:35.183 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.180+0000 7f8f408b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:34:35.183 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.180+0000 7f8f408b3640 1 -- 172.21.15.135:0/1958589974 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f3c148a00 con 0x7f8f3c14a700 2024-03-29T11:34:35.183 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.181+0000 7f8f408b3640 10 client.4566.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:35.183 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.181+0000 7f8f408b3640 10 monclient: _renew_subs 2024-03-29T11:34:35.183 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.181+0000 7f8f408b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:34:35.183 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.181+0000 7f8f408b3640 1 -- 172.21.15.135:0/1958589974 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f3c1235c0 con 0x7f8f3c14a700 2024-03-29T11:34:35.183 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.181+0000 7f8f408b3640 1 librados: init done 2024-03-29T11:34:35.183 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.183+0000 7f8f408b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:35.184 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.184+0000 7f8f408b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:35.184 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.184+0000 7f8f408b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:34:35.184 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.184+0000 7f8f2affd640 1 -- 172.21.15.135:0/1958589974 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f8f3401b0e0 con 0x7f8f3c14a700 2024-03-29T11:34:35.184 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.184+0000 7f8f408b3640 1 -- 172.21.15.135:0/1958589974 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f3c0ab560 con 0x7f8f3c14a700 2024-03-29T11:34:35.184 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.185+0000 7f8f2affd640 1 -- 172.21.15.135:0/1958589974 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1288947923 0 0) 0x7f8f34084da0 con 0x7f8f3c14a700 2024-03-29T11:34:35.184 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.185+0000 7f8f2affd640 10 client.4566.objecter ms_dispatch 0x7f8f3c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:34:35.184 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.185+0000 7f8f2affd640 3 client.4566.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:34:35.184 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.185+0000 7f8f2affd640 3 client.4566.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:34:35.184 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.185+0000 7f8f2affd640 20 client.4566.objecter dump_active .. 0 homeless 2024-03-29T11:34:35.186 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.187+0000 7f8f3a59c640 10 monclient: get_auth_request con 0x7f8f1c06de80 auth_method 0 2024-03-29T11:34:35.186 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.188+0000 7f8f2affd640 10 client.4566.objecter ms_handle_connect 0x7f8f1c06de80 2024-03-29T11:34:35.194 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.196+0000 7f8f2affd640 1 -- 172.21.15.135:0/1958589974 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8f34050680 con 0x7f8f3c14a700 2024-03-29T11:34:35.195 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.197+0000 7f8f2affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:35.195 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.197+0000 7f8f2affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:35.514 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.517+0000 7f8f408b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:35.516 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.517+0000 7f8f408b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:35.517 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.517+0000 7f8f408b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:34:35.517 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.517+0000 7f8f408b3640 1 -- 172.21.15.135:0/1958589974 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f3c1215e0 con 0x7f8f3c14a700 2024-03-29T11:34:35.518 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.520+0000 7f8f2affd640 1 -- 172.21.15.135:0/1958589974 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2351702130) 0x7f8f340525b0 con 0x7f8f3c14a700 2024-03-29T11:34:35.518 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.520+0000 7f8f2affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:35.519 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.520+0000 7f8f2affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:35.519 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:34:35.519 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":11,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:34:26.903077+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:26.903077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:35.524 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.524+0000 7f8f408b3640 10 librados: watch_flush enter 2024-03-29T11:34:35.524 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.525+0000 7f8f408b3640 10 librados: watch_flush exit 2024-03-29T11:34:35.524 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.525+0000 7f8f408b3640 20 client.4566.objecter shutdown clearing up homeless session... 2024-03-29T11:34:35.524 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.525+0000 7f8f408b3640 10 client.4566.objecter successfully canceled tick 2024-03-29T11:34:35.524 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.525+0000 7f8f408b3640 1 -- 172.21.15.135:0/1958589974 >> v1:172.21.15.135:6816/3469875285 conn(0x7f8f1c06de80 legacy=0x7f8f1c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:35.524 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.525+0000 7f8f408b3640 10 monclient: shutdown 2024-03-29T11:34:35.524 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.525+0000 7f8f408b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:35.524 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.525+0000 7f8f408b3640 1 -- 172.21.15.135:0/1958589974 >> v1:172.21.15.135:6789/0 conn(0x7f8f3c14a700 legacy=0x7f8f3c1e3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:35.524 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.525+0000 7f8f408b3640 1 -- 172.21.15.135:0/1958589974 shutdown_connections 2024-03-29T11:34:35.524 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.525+0000 7f8f408b3640 1 -- 172.21.15.135:0/1958589974 >> 172.21.15.135:0/1958589974 conn(0x7f8f3c06b660 msgr2=0x7f8f3c149510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:35.524 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.526+0000 7f8f408b3640 1 -- 172.21.15.135:0/1958589974 shutdown_connections 2024-03-29T11:34:35.525 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.526+0000 7f8f408b3640 1 -- 172.21.15.135:0/1958589974 wait complete. 2024-03-29T11:34:35.525 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:35.527+0000 7f8f408b3640 1 librados: shutdown 2024-03-29T11:34:35.544 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:42.350 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:42.517 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.518+0000 7fafc6d67640 10 monclient: get_monmap_and_config 2024-03-29T11:34:42.517 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.518+0000 7fafc6d67640 10 monclient: build_initial_monmap 2024-03-29T11:34:42.517 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.518+0000 7fafc6d67640 10 monclient: monmap: 2024-03-29T11:34:42.517 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:34:42.517 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:42.517 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:34:42.518811+0000 2024-03-29T11:34:42.517 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:34:42.518811+0000 2024-03-29T11:34:42.517 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:42.517 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:42.517 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:34:42.517 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:34:42.518 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:34:42.518 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:42.518 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.521+0000 7fafc6d67640 1 Processor -- start 2024-03-29T11:34:42.519 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.522+0000 7fafc6d67640 1 -- start start 2024-03-29T11:34:42.519 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.522+0000 7fafc6d67640 10 monclient: init 2024-03-29T11:34:42.520 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.523+0000 7fafc6d67640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:42.520 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.523+0000 7fafc6d67640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:34:42.520 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.523+0000 7fafc6d67640 10 monclient(hunting): picked mon.noname-c con 0x7fafc00a9ac0 addr v1:172.21.15.194:6789/0 2024-03-29T11:34:42.520 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.523+0000 7fafc6d67640 10 monclient(hunting): picked mon.noname-a con 0x7fafc013ee90 addr v1:172.21.15.135:6789/0 2024-03-29T11:34:42.520 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.523+0000 7fafc6d67640 10 monclient(hunting): picked mon.noname-b con 0x7fafc00af900 addr v1:172.21.15.135:6790/0 2024-03-29T11:34:42.521 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.523+0000 7fafc6d67640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc00afd10 con 0x7fafc013ee90 2024-03-29T11:34:42.521 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.523+0000 7fafc6d67640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc00af6f0 con 0x7fafc00a9ac0 2024-03-29T11:34:42.521 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.523+0000 7fafc6d67640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc00aac80 con 0x7fafc00af900 2024-03-29T11:34:42.521 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.523+0000 7fafc6d67640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:42.521 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.523+0000 7fafc6d67640 10 monclient(hunting): authenticate will time out at 1389.891258s 2024-03-29T11:34:42.526 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.526+0000 7fafc5304640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7fafc00af900 0x7fafc0149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:55983/0 (socket says 172.21.15.135:55983) 2024-03-29T11:34:42.526 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.526+0000 7fafc5304640 1 -- 172.21.15.135:0/507229628 learned_addr learned my addr 172.21.15.135:0/507229628 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:34:42.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.530+0000 7fafbf7fe640 1 -- 172.21.15.135:0/507229628 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284870795 0 0) 0x7fafc00af6f0 con 0x7fafc00a9ac0 2024-03-29T11:34:42.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.530+0000 7fafbf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:42.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.530+0000 7fafbf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:42.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.530+0000 7fafbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:42.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.530+0000 7fafbf7fe640 10 monclient(hunting): my global_id is 4591 2024-03-29T11:34:42.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.530+0000 7fafbf7fe640 1 -- 172.21.15.135:0/507229628 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafa8003520 con 0x7fafc00a9ac0 2024-03-29T11:34:42.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.531+0000 7fafbf7fe640 1 -- 172.21.15.135:0/507229628 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2886692604 0 0) 0x7fafa8003520 con 0x7fafc00a9ac0 2024-03-29T11:34:42.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.531+0000 7fafbf7fe640 1 -- 172.21.15.135:0/507229628 >> v1:172.21.15.135:6790/0 conn(0x7fafc00af900 legacy=0x7fafc0149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:42.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.531+0000 7fafbf7fe640 1 -- 172.21.15.135:0/507229628 >> v1:172.21.15.135:6789/0 conn(0x7fafc013ee90 legacy=0x7fafc013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:42.530 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.531+0000 7fafbf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:34:42.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.531+0000 7fafbf7fe640 1 monclient: found mon.noname-c 2024-03-29T11:34:42.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.531+0000 7fafbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:42.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.531+0000 7fafbf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:34:42.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.531+0000 7fafbf7fe640 1 -- 172.21.15.135:0/507229628 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc00aaf60 con 0x7fafc00a9ac0 2024-03-29T11:34:42.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.531+0000 7fafbf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:34:42.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.531+0000 7fafbf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:34:42.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.531+0000 7fafbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:42.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.531+0000 7fafc6d67640 5 monclient: authenticate success, global_id 4591 2024-03-29T11:34:42.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.531+0000 7fafc6d67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:42.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.531+0000 7fafbf7fe640 1 -- 172.21.15.135:0/507229628 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fafb4001690 con 0x7fafc00a9ac0 2024-03-29T11:34:42.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.531+0000 7fafbf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:42.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.531+0000 7fafbf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:42.531 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.531+0000 7fafbf7fe640 10 monclient: dump: 2024-03-29T11:34:42.531 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:42.531 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:42.532 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:42.532 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:42.532 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:42.532 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:42.532 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:42.532 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:42.532 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:42.532 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:42.532 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.531+0000 7fafc6d67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:42.532 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.535+0000 7fafbf7fe640 1 -- 172.21.15.135:0/507229628 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafb40017f0 con 0x7fafc00a9ac0 2024-03-29T11:34:42.532 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.535+0000 7fafbf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:42.533 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.535+0000 7fafc6d67640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:42.533 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.535+0000 7fafc6d67640 4 set_mon_vals no callback set 2024-03-29T11:34:42.535 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.535+0000 7fafc6d67640 10 monclient: shutdown 2024-03-29T11:34:42.535 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.535+0000 7fafc6d67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:42.535 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.535+0000 7fafc6d67640 1 -- 172.21.15.135:0/507229628 >> v1:172.21.15.194:6789/0 conn(0x7fafc00a9ac0 legacy=0x7fafc00adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:42.536 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.539+0000 7fafc6d67640 1 -- 172.21.15.135:0/507229628 shutdown_connections 2024-03-29T11:34:42.536 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.539+0000 7fafc6d67640 1 -- 172.21.15.135:0/507229628 >> 172.21.15.135:0/507229628 conn(0x7fafc00b35e0 msgr2=0x7fafc00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:42.537 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.540+0000 7fafc6d67640 1 -- 172.21.15.135:0/507229628 shutdown_connections 2024-03-29T11:34:42.538 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.541+0000 7fafc6d67640 1 -- 172.21.15.135:0/507229628 wait complete. 2024-03-29T11:34:42.539 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.542+0000 7fafc6d67640 10 monclient: build_initial_monmap 2024-03-29T11:34:42.539 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.542+0000 7fafc6d67640 10 monclient: monmap: 2024-03-29T11:34:42.539 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:34:42.539 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:42.539 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:34:42.539 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:34:42.540 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:42.540 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:42.540 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:34:42.540 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:34:42.540 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:34:42.540 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:42.540 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.542+0000 7fafc6d67640 1 librados: starting msgr at 2024-03-29T11:34:42.540 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.542+0000 7fafc6d67640 1 librados: starting objecter 2024-03-29T11:34:42.540 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.542+0000 7fafc6d67640 1 Processor -- start 2024-03-29T11:34:42.540 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.542+0000 7fafc6d67640 1 -- start start 2024-03-29T11:34:42.541 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.542+0000 7fafc6d67640 1 librados: setting wanted keys 2024-03-29T11:34:42.541 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.542+0000 7fafc6d67640 1 librados: calling monclient init 2024-03-29T11:34:42.541 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.542+0000 7fafc6d67640 10 monclient: init 2024-03-29T11:34:42.547 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.543+0000 7fafc6d67640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:42.547 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.543+0000 7fafc6d67640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:34:42.547 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.543+0000 7fafc6d67640 10 monclient(hunting): picked mon.noname-c con 0x7fafc00a9ac0 addr v1:172.21.15.135:6790/0 2024-03-29T11:34:42.548 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.543+0000 7fafc6d67640 10 monclient(hunting): picked mon.noname-b con 0x7fafc00af900 addr v1:172.21.15.194:6789/0 2024-03-29T11:34:42.548 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.543+0000 7fafc6d67640 10 monclient(hunting): picked mon.noname-a con 0x7fafc013ee90 addr v1:172.21.15.135:6789/0 2024-03-29T11:34:42.548 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.543+0000 7fafc6d67640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc0071ad0 con 0x7fafc013ee90 2024-03-29T11:34:42.548 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.543+0000 7fafc6d67640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc0071d90 con 0x7fafc00af900 2024-03-29T11:34:42.548 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.543+0000 7fafc6d67640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc0135170 con 0x7fafc00a9ac0 2024-03-29T11:34:42.548 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.543+0000 7fafc6d67640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:42.548 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.543+0000 7fafc6d67640 10 monclient(hunting): authenticate will time out at 1389.911677s 2024-03-29T11:34:42.550 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.550+0000 7fafc5304640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fafc013ee90 0x7fafc0134a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:52667/0 (socket says 172.21.15.135:52667) 2024-03-29T11:34:42.550 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.550+0000 7fafc5304640 1 -- 172.21.15.135:0/4108849916 learned_addr learned my addr 172.21.15.135:0/4108849916 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:34:42.552 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.553+0000 7fafbd7fa640 10 client.?.objecter ms_handle_connect 0x7fafc013ee90 2024-03-29T11:34:42.552 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.553+0000 7fafbd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:42.552 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.553+0000 7fafbd7fa640 10 client.?.objecter ms_handle_connect 0x7fafc00a9ac0 2024-03-29T11:34:42.552 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.553+0000 7fafbd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:42.552 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.553+0000 7fafbd7fa640 1 -- 172.21.15.135:0/4108849916 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950490494 0 0) 0x7fafc0071ad0 con 0x7fafc013ee90 2024-03-29T11:34:42.552 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.553+0000 7fafbd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:34:42.552 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.553+0000 7fafbd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:42.552 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.553+0000 7fafbd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:42.556 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.553+0000 7fafbd7fa640 10 monclient(hunting): my global_id is 4572 2024-03-29T11:34:42.557 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.553+0000 7fafbd7fa640 1 -- 172.21.15.135:0/4108849916 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafac004eb0 con 0x7fafc013ee90 2024-03-29T11:34:42.557 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.553+0000 7fafbd7fa640 1 -- 172.21.15.135:0/4108849916 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 119032880 0 0) 0x7fafc0135170 con 0x7fafc00a9ac0 2024-03-29T11:34:42.557 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.553+0000 7fafbd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:34:42.557 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.553+0000 7fafbd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:42.557 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.553+0000 7fafbd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:42.557 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.553+0000 7fafbd7fa640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:34:42.557 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.553+0000 7fafbd7fa640 1 -- 172.21.15.135:0/4108849916 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafc0071ad0 con 0x7fafc00a9ac0 2024-03-29T11:34:42.557 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.555+0000 7fafbd7fa640 1 -- 172.21.15.135:0/4108849916 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1038602205 0 0) 0x7fafc0071ad0 con 0x7fafc00a9ac0 2024-03-29T11:34:42.557 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.555+0000 7fafbd7fa640 1 -- 172.21.15.135:0/4108849916 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafc0135170 con 0x7fafc00a9ac0 2024-03-29T11:34:42.557 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.555+0000 7fafbd7fa640 10 client.?.objecter ms_handle_connect 0x7fafc00af900 2024-03-29T11:34:42.557 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.555+0000 7fafbd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:42.557 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.555+0000 7fafbd7fa640 1 -- 172.21.15.135:0/4108849916 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fafb4003850 con 0x7fafc00a9ac0 2024-03-29T11:34:42.557 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.555+0000 7fafbd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:42.558 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.555+0000 7fafbd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:42.558 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.555+0000 7fafbd7fa640 10 monclient(hunting): dump: 2024-03-29T11:34:42.558 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:42.558 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:42.558 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:42.558 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:42.558 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:42.558 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:42.558 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:42.558 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:42.558 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:42.558 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:42.558 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.555+0000 7fafbd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:42.559 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.555+0000 7fafbd7fa640 1 -- 172.21.15.135:0/4108849916 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2378788730 0 0) 0x7fafac004eb0 con 0x7fafc013ee90 2024-03-29T11:34:42.559 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.555+0000 7fafbd7fa640 1 -- 172.21.15.135:0/4108849916 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafac006a40 con 0x7fafc013ee90 2024-03-29T11:34:42.559 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.555+0000 7fafbd7fa640 1 -- 172.21.15.135:0/4108849916 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572248684 0 0) 0x7fafc0071d90 con 0x7fafc00af900 2024-03-29T11:34:42.559 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.555+0000 7fafbd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:34:42.559 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.556+0000 7fafbd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:42.559 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.556+0000 7fafbd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:42.559 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.556+0000 7fafbd7fa640 10 monclient(hunting): my global_id is 4594 2024-03-29T11:34:42.559 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.556+0000 7fafbd7fa640 1 -- 172.21.15.135:0/4108849916 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafac004eb0 con 0x7fafc00af900 2024-03-29T11:34:42.559 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.556+0000 7fafbd7fa640 1 -- 172.21.15.135:0/4108849916 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fafb0005e70 con 0x7fafc013ee90 2024-03-29T11:34:42.559 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.556+0000 7fafbd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:42.559 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.556+0000 7fafbd7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:42.559 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.556+0000 7fafbd7fa640 10 monclient(hunting): dump: 2024-03-29T11:34:42.559 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:42.560 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:42.560 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:42.560 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:42.560 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:42.560 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:42.560 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:42.560 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:42.560 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:42.560 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:42.560 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.557+0000 7fafbd7fa640 1 -- 172.21.15.135:0/4108849916 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1805153617 0 0) 0x7fafac006a40 con 0x7fafc013ee90 2024-03-29T11:34:42.560 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.557+0000 7fafbd7fa640 1 -- 172.21.15.135:0/4108849916 >> v1:172.21.15.135:6790/0 conn(0x7fafc00a9ac0 legacy=0x7fafc01dc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:42.560 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.557+0000 7fafbd7fa640 1 -- 172.21.15.135:0/4108849916 >> v1:172.21.15.194:6789/0 conn(0x7fafc00af900 legacy=0x7fafc0134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:42.560 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.557+0000 7fafbd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:34:42.560 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.557+0000 7fafbd7fa640 1 monclient: found mon.a 2024-03-29T11:34:42.560 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.557+0000 7fafbd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:42.561 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.557+0000 7fafbd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:34:42.561 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.557+0000 7fafbd7fa640 1 -- 172.21.15.135:0/4108849916 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc01dc8a0 con 0x7fafc013ee90 2024-03-29T11:34:42.561 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.557+0000 7fafbd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:34:42.561 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.557+0000 7fafbd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:34:42.561 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.557+0000 7fafbd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:42.561 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.559+0000 7fafc6d67640 5 monclient: authenticate success, global_id 4572 2024-03-29T11:34:42.561 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.559+0000 7fafc6d67640 10 monclient: _renew_subs 2024-03-29T11:34:42.561 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.559+0000 7fafc6d67640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:34:42.561 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.559+0000 7fafc6d67640 1 -- 172.21.15.135:0/4108849916 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc0136e30 con 0x7fafc013ee90 2024-03-29T11:34:42.561 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.559+0000 7fafc6d67640 10 client.4572.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:42.561 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.559+0000 7fafc6d67640 10 monclient: _renew_subs 2024-03-29T11:34:42.561 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.559+0000 7fafc6d67640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:34:42.561 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.559+0000 7fafc6d67640 1 -- 172.21.15.135:0/4108849916 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc0135870 con 0x7fafc013ee90 2024-03-29T11:34:42.561 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.559+0000 7fafc6d67640 1 librados: init done 2024-03-29T11:34:42.563 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.561+0000 7fafc6d67640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:42.563 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.561+0000 7fafc6d67640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:42.563 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.561+0000 7fafc6d67640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:34:42.563 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.561+0000 7fafc6d67640 1 -- 172.21.15.135:0/4108849916 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc006dfd0 con 0x7fafc013ee90 2024-03-29T11:34:42.563 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.561+0000 7fafbd7fa640 1 -- 172.21.15.135:0/4108849916 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafb0001520 con 0x7fafc013ee90 2024-03-29T11:34:42.563 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.561+0000 7fafbd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:42.563 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.561+0000 7fafbd7fa640 1 -- 172.21.15.135:0/4108849916 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fafb0001930 con 0x7fafc013ee90 2024-03-29T11:34:42.563 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.561+0000 7fafbd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:42.563 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.561+0000 7fafbd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:42.563 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.561+0000 7fafbd7fa640 10 monclient: dump: 2024-03-29T11:34:42.564 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:42.564 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:42.564 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:42.564 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:42.564 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:42.564 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:42.564 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:42.564 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:42.564 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:42.564 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:42.564 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.561+0000 7fafbeffd640 4 set_mon_vals no callback set 2024-03-29T11:34:42.564 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.563+0000 7fafbd7fa640 1 -- 172.21.15.135:0/4108849916 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fafb0001b50 con 0x7fafc013ee90 2024-03-29T11:34:42.564 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.564+0000 7fafbd7fa640 1 -- 172.21.15.135:0/4108849916 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1288947923 0 0) 0x7fafb0084a10 con 0x7fafc013ee90 2024-03-29T11:34:42.564 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.564+0000 7fafbd7fa640 10 client.4572.objecter ms_dispatch 0x7fafc0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:34:42.564 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.564+0000 7fafbd7fa640 3 client.4572.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:34:42.565 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.564+0000 7fafbd7fa640 3 client.4572.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:34:42.565 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.564+0000 7fafbd7fa640 20 client.4572.objecter dump_active .. 0 homeless 2024-03-29T11:34:42.565 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.565+0000 7fafc4b03640 10 monclient: get_auth_request con 0x7fafac06e440 auth_method 0 2024-03-29T11:34:42.565 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.565+0000 7fafbd7fa640 10 client.4572.objecter ms_handle_connect 0x7fafac06e440 2024-03-29T11:34:42.571 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.574+0000 7fafbd7fa640 1 -- 172.21.15.135:0/4108849916 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fafb00502f0 con 0x7fafc013ee90 2024-03-29T11:34:42.571 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.574+0000 7fafbd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:42.572 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.574+0000 7fafbd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:42.893 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.895+0000 7fafc6d67640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:42.893 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.895+0000 7fafc6d67640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:42.893 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.895+0000 7fafc6d67640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:34:42.893 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.895+0000 7fafc6d67640 1 -- 172.21.15.135:0/4108849916 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fafc0075f10 con 0x7fafc013ee90 2024-03-29T11:34:42.896 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.899+0000 7fafbd7fa640 1 -- 172.21.15.135:0/4108849916 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2351702130) 0x7fafb0052220 con 0x7fafc013ee90 2024-03-29T11:34:42.897 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.899+0000 7fafbd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:42.897 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.899+0000 7fafbd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:42.897 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:34:42.897 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":11,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:34:26.903077+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:26.903077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:42.900 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.903+0000 7fafc6d67640 10 librados: watch_flush enter 2024-03-29T11:34:42.900 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.903+0000 7fafc6d67640 10 librados: watch_flush exit 2024-03-29T11:34:42.900 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.903+0000 7fafc6d67640 20 client.4572.objecter shutdown clearing up homeless session... 2024-03-29T11:34:42.900 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.903+0000 7fafc6d67640 10 client.4572.objecter successfully canceled tick 2024-03-29T11:34:42.901 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.903+0000 7fafc6d67640 1 -- 172.21.15.135:0/4108849916 >> v1:172.21.15.135:6816/3469875285 conn(0x7fafac06e440 legacy=0x7fafac070900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:42.901 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.903+0000 7fafc6d67640 10 monclient: shutdown 2024-03-29T11:34:42.901 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.903+0000 7fafc6d67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:42.901 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.903+0000 7fafc6d67640 1 -- 172.21.15.135:0/4108849916 >> v1:172.21.15.135:6789/0 conn(0x7fafc013ee90 legacy=0x7fafc0134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:42.901 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.903+0000 7fafc6d67640 1 -- 172.21.15.135:0/4108849916 shutdown_connections 2024-03-29T11:34:42.901 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.903+0000 7fafc6d67640 1 -- 172.21.15.135:0/4108849916 >> 172.21.15.135:0/4108849916 conn(0x7fafc00b35e0 msgr2=0x7fafc00adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:42.902 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.904+0000 7fafc6d67640 1 -- 172.21.15.135:0/4108849916 shutdown_connections 2024-03-29T11:34:42.902 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.904+0000 7fafc6d67640 1 -- 172.21.15.135:0/4108849916 wait complete. 2024-03-29T11:34:42.902 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:42.904+0000 7fafc6d67640 1 librados: shutdown 2024-03-29T11:34:42.920 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:49.727 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:49.878 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.879+0000 7f51523c6640 10 monclient: get_monmap_and_config 2024-03-29T11:34:49.878 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.879+0000 7f51523c6640 10 monclient: build_initial_monmap 2024-03-29T11:34:49.880 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.879+0000 7f51523c6640 10 monclient: monmap: 2024-03-29T11:34:49.880 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:34:49.881 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:49.881 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:34:49.880015+0000 2024-03-29T11:34:49.881 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:34:49.880015+0000 2024-03-29T11:34:49.881 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:49.881 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:49.882 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:34:49.882 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:34:49.882 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:34:49.882 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:49.882 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.880+0000 7f51523c6640 1 Processor -- start 2024-03-29T11:34:49.882 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.880+0000 7f51523c6640 1 -- start start 2024-03-29T11:34:49.883 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.880+0000 7f51523c6640 10 monclient: init 2024-03-29T11:34:49.883 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.881+0000 7f51523c6640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:49.883 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.881+0000 7f51523c6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:34:49.883 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.881+0000 7f51523c6640 10 monclient(hunting): picked mon.noname-b con 0x7f514c0a9930 addr v1:172.21.15.135:6790/0 2024-03-29T11:34:49.883 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.881+0000 7f51523c6640 10 monclient(hunting): picked mon.noname-a con 0x7f514c149dc0 addr v1:172.21.15.135:6789/0 2024-03-29T11:34:49.884 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.881+0000 7f51523c6640 10 monclient(hunting): picked mon.noname-c con 0x7f514c14ff40 addr v1:172.21.15.194:6789/0 2024-03-29T11:34:49.884 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.881+0000 7f51523c6640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f514c0aae60 con 0x7f514c149dc0 2024-03-29T11:34:49.884 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.881+0000 7f51523c6640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f514c0b4310 con 0x7f514c14ff40 2024-03-29T11:34:49.884 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.881+0000 7f51523c6640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f514c0afdd0 con 0x7f514c0a9930 2024-03-29T11:34:49.884 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.881+0000 7f51523c6640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:49.884 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.881+0000 7f51523c6640 10 monclient(hunting): authenticate will time out at 1397.249749s 2024-03-29T11:34:49.885 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.882+0000 7f514b7fe640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f514c149dc0 0x7f514c14a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:59024/0 (socket says 172.21.15.135:59024) 2024-03-29T11:34:49.885 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.882+0000 7f514b7fe640 1 -- 172.21.15.135:0/3527520482 learned_addr learned my addr 172.21.15.135:0/3527520482 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:34:49.885 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.883+0000 7f514affd640 1 -- 172.21.15.135:0/3527520482 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1783256442 0 0) 0x7f514c0aae60 con 0x7f514c149dc0 2024-03-29T11:34:49.885 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.883+0000 7f514affd640 10 monclient(hunting): _negotiate 2024-03-29T11:34:49.885 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.883+0000 7f514affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:49.886 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.883+0000 7f514affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:49.886 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.883+0000 7f514affd640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:34:49.886 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.883+0000 7f514affd640 1 -- 172.21.15.135:0/3527520482 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5138003520 con 0x7f514c149dc0 2024-03-29T11:34:49.886 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.883+0000 7f514affd640 1 -- 172.21.15.135:0/3527520482 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3135902186 0 0) 0x7f5138003520 con 0x7f514c149dc0 2024-03-29T11:34:49.886 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.883+0000 7f514affd640 1 -- 172.21.15.135:0/3527520482 >> v1:172.21.15.135:6790/0 conn(0x7f514c0a9930 legacy=0x7f514c143a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:34:49.889 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.885+0000 7f514affd640 1 -- 172.21.15.135:0/3527520482 >> v1:172.21.15.194:6789/0 conn(0x7f514c14ff40 legacy=0x7f514c152330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:34:49.889 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.885+0000 7f514affd640 10 monclient: _finish_hunting 0 2024-03-29T11:34:49.889 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.885+0000 7f514affd640 1 monclient: found mon.noname-a 2024-03-29T11:34:49.890 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.885+0000 7f514affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:49.890 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.885+0000 7f514affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.135:6789/0 2024-03-29T11:34:49.890 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.885+0000 7f514affd640 1 -- 172.21.15.135:0/3527520482 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514c14a950 con 0x7f514c149dc0 2024-03-29T11:34:49.890 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.885+0000 7f514affd640 10 monclient: _finish_auth 0 2024-03-29T11:34:49.890 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.885+0000 7f514affd640 10 monclient: _check_auth_tickets 2024-03-29T11:34:49.890 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.885+0000 7f514affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:49.891 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.885+0000 7f514affd640 1 -- 172.21.15.135:0/3527520482 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f513c001670 con 0x7f514c149dc0 2024-03-29T11:34:49.891 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.885+0000 7f514affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:49.891 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.885+0000 7f514affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:49.891 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.885+0000 7f514affd640 10 monclient: dump: 2024-03-29T11:34:49.891 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:49.891 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:49.892 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:49.892 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:49.892 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:49.892 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:49.892 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:49.892 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:49.892 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:49.893 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:49.893 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.885+0000 7f51523c6640 5 monclient: authenticate success, global_id 4575 2024-03-29T11:34:49.893 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.885+0000 7f51523c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:34:49.893 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.885+0000 7f514affd640 1 -- 172.21.15.135:0/3527520482 <== mon.0 v1:172.21.15.135:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f513c0018d0 con 0x7f514c149dc0 2024-03-29T11:34:49.893 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.885+0000 7f514affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:49.893 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.885+0000 7f514affd640 1 -- 172.21.15.135:0/3527520482 <== mon.0 v1:172.21.15.135:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f513c001d20 con 0x7f514c149dc0 2024-03-29T11:34:49.893 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.885+0000 7f514affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:49.893 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.885+0000 7f514affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:49.894 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.885+0000 7f514affd640 10 monclient: dump: 2024-03-29T11:34:49.894 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:49.894 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:49.894 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:49.894 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:49.894 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:49.894 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:49.895 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:49.895 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:49.895 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:49.895 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:49.895 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.886+0000 7f51523c6640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:49.895 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.886+0000 7f51523c6640 4 set_mon_vals no callback set 2024-03-29T11:34:49.895 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.886+0000 7f51523c6640 10 monclient: shutdown 2024-03-29T11:34:49.895 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.887+0000 7f51523c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:49.896 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.887+0000 7f51523c6640 1 -- 172.21.15.135:0/3527520482 >> v1:172.21.15.135:6789/0 conn(0x7f514c149dc0 legacy=0x7f514c14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:49.896 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.889+0000 7f51523c6640 1 -- 172.21.15.135:0/3527520482 shutdown_connections 2024-03-29T11:34:49.896 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.889+0000 7f51523c6640 1 -- 172.21.15.135:0/3527520482 >> 172.21.15.135:0/3527520482 conn(0x7f514c0b35e0 msgr2=0x7f514c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:49.903 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.893+0000 7f51523c6640 1 -- 172.21.15.135:0/3527520482 shutdown_connections 2024-03-29T11:34:49.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.893+0000 7f51523c6640 1 -- 172.21.15.135:0/3527520482 wait complete. 2024-03-29T11:34:49.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.894+0000 7f51523c6640 10 monclient: build_initial_monmap 2024-03-29T11:34:49.904 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.894+0000 7f51523c6640 10 monclient: monmap: 2024-03-29T11:34:49.904 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:34:49.904 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:49.904 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:34:49.904 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:34:49.904 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:49.905 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:49.905 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:34:49.905 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:34:49.905 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:34:49.905 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:49.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.894+0000 7f51523c6640 1 librados: starting msgr at 2024-03-29T11:34:49.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.894+0000 7f51523c6640 1 librados: starting objecter 2024-03-29T11:34:49.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.895+0000 7f51523c6640 1 Processor -- start 2024-03-29T11:34:49.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.895+0000 7f51523c6640 1 -- start start 2024-03-29T11:34:49.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.895+0000 7f51523c6640 1 librados: setting wanted keys 2024-03-29T11:34:49.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.895+0000 7f51523c6640 1 librados: calling monclient init 2024-03-29T11:34:49.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.895+0000 7f51523c6640 10 monclient: init 2024-03-29T11:34:49.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.896+0000 7f51523c6640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:49.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.896+0000 7f51523c6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:34:49.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.896+0000 7f51523c6640 10 monclient(hunting): picked mon.noname-a con 0x7f514c0a9930 addr v1:172.21.15.135:6789/0 2024-03-29T11:34:49.905 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.897+0000 7f514bfff640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7f514c0a9930 0x7f514c12eb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:41958/0 (socket says 172.21.15.135:41958) 2024-03-29T11:34:49.906 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.899+0000 7f514bfff640 1 -- 172.21.15.135:0/1755690240 learned_addr learned my addr 172.21.15.135:0/1755690240 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:34:49.906 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.899+0000 7f51523c6640 10 monclient(hunting): picked mon.noname-b con 0x7f514c149dc0 addr v1:172.21.15.194:6789/0 2024-03-29T11:34:49.906 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.899+0000 7f5148ff9640 10 client.?.objecter ms_handle_connect 0x7f514c0a9930 2024-03-29T11:34:49.906 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.899+0000 7f5148ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:49.906 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.899+0000 7f51523c6640 10 monclient(hunting): picked mon.noname-c con 0x7f514c14ff40 addr v1:172.21.15.135:6790/0 2024-03-29T11:34:49.906 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.900+0000 7f51523c6640 1 -- 172.21.15.135:0/1755690240 --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f514c14aae0 con 0x7f514c0a9930 2024-03-29T11:34:49.906 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.900+0000 7f5148ff9640 10 client.?.objecter ms_handle_connect 0x7f514c149dc0 2024-03-29T11:34:49.906 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.900+0000 7f5148ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:49.906 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.900+0000 7f51523c6640 1 -- 172.21.15.135:0/1755690240 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f514c14b230 con 0x7f514c149dc0 2024-03-29T11:34:49.906 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.900+0000 7f51523c6640 1 -- 172.21.15.135:0/1755690240 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f514c14b510 con 0x7f514c14ff40 2024-03-29T11:34:49.906 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.900+0000 7f51523c6640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:49.906 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.900+0000 7f51523c6640 10 monclient(hunting): authenticate will time out at 1397.268352s 2024-03-29T11:34:49.906 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.900+0000 7f5148ff9640 1 -- 172.21.15.135:0/1755690240 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830433476 0 0) 0x7f514c14aae0 con 0x7f514c0a9930 2024-03-29T11:34:49.906 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.900+0000 7f5148ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:34:49.907 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.900+0000 7f5148ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:49.907 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.900+0000 7f5148ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:49.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.900+0000 7f5148ff9640 10 monclient(hunting): my global_id is 4578 2024-03-29T11:34:49.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.900+0000 7f5148ff9640 1 -- 172.21.15.135:0/1755690240 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f512c004eb0 con 0x7f514c0a9930 2024-03-29T11:34:49.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.901+0000 7f5148ff9640 1 -- 172.21.15.135:0/1755690240 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 420076100 0 0) 0x7f512c004eb0 con 0x7f514c0a9930 2024-03-29T11:34:49.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.901+0000 7f5148ff9640 1 -- 172.21.15.135:0/1755690240 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f514c14aae0 con 0x7f514c0a9930 2024-03-29T11:34:49.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.901+0000 7f5148ff9640 1 -- 172.21.15.135:0/1755690240 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f5144001ce0 con 0x7f514c0a9930 2024-03-29T11:34:49.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.901+0000 7f5148ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:49.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.901+0000 7f5148ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:34:49.908 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.901+0000 7f5148ff9640 10 monclient(hunting): dump: 2024-03-29T11:34:49.908 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:49.908 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:49.908 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:49.908 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:49.908 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:49.908 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:49.909 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:49.909 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:49.909 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:49.909 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:49.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.902+0000 7f5148ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:49.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.902+0000 7f5148ff9640 1 -- 172.21.15.135:0/1755690240 <== mon.0 v1:172.21.15.135:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1143260447 0 0) 0x7f514c14aae0 con 0x7f514c0a9930 2024-03-29T11:34:49.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.902+0000 7f5148ff9640 1 -- 172.21.15.135:0/1755690240 >> v1:172.21.15.135:6790/0 conn(0x7f514c14ff40 legacy=0x7f514c1de090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:34:49.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.903+0000 7f5148ff9640 1 -- 172.21.15.135:0/1755690240 >> v1:172.21.15.194:6789/0 conn(0x7f514c149dc0 legacy=0x7f514c06e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:49.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.903+0000 7f5148ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:34:49.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.903+0000 7f5148ff9640 1 monclient: found mon.a 2024-03-29T11:34:49.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.903+0000 7f5148ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:49.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.903+0000 7f5148ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:34:49.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.903+0000 7f5148ff9640 1 -- 172.21.15.135:0/1755690240 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514c130dc0 con 0x7f514c0a9930 2024-03-29T11:34:49.909 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.903+0000 7f5148ff9640 10 monclient: _finish_auth 0 2024-03-29T11:34:49.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.903+0000 7f5148ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:34:49.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.903+0000 7f5148ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:49.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.903+0000 7f5148ff9640 1 -- 172.21.15.135:0/1755690240 <== mon.0 v1:172.21.15.135:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51440020f0 con 0x7f514c0a9930 2024-03-29T11:34:49.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.903+0000 7f5148ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:49.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.903+0000 7f514a7fc640 4 set_mon_vals no callback set 2024-03-29T11:34:49.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.904+0000 7f51523c6640 5 monclient: authenticate success, global_id 4578 2024-03-29T11:34:49.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.904+0000 7f51523c6640 10 monclient: _renew_subs 2024-03-29T11:34:49.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.904+0000 7f51523c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:34:49.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.904+0000 7f51523c6640 1 -- 172.21.15.135:0/1755690240 --> v1:172.21.15.135:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f514c071e30 con 0x7f514c0a9930 2024-03-29T11:34:49.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.905+0000 7f51523c6640 10 client.4578.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:49.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.905+0000 7f51523c6640 10 monclient: _renew_subs 2024-03-29T11:34:49.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.905+0000 7f51523c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:34:49.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.905+0000 7f51523c6640 1 -- 172.21.15.135:0/1755690240 --> v1:172.21.15.135:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f514c0709a0 con 0x7f514c0a9930 2024-03-29T11:34:49.910 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.905+0000 7f51523c6640 1 librados: init done 2024-03-29T11:34:49.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.905+0000 7f5148ff9640 1 -- 172.21.15.135:0/1755690240 <== mon.0 v1:172.21.15.135:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f5144001f00 con 0x7f514c0a9930 2024-03-29T11:34:49.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.905+0000 7f5148ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:49.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.905+0000 7f5148ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:34:49.911 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.905+0000 7f5148ff9640 10 monclient: dump: 2024-03-29T11:34:49.911 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:49.911 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:49.911 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:49.911 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:49.911 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:49.911 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:49.911 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:49.911 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:49.911 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:49.911 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:49.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.907+0000 7f51523c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:49.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.907+0000 7f51523c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:49.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.907+0000 7f51523c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:34:49.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.907+0000 7f51523c6640 1 -- 172.21.15.135:0/1755690240 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f514c075f10 con 0x7f514c0a9930 2024-03-29T11:34:49.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.907+0000 7f5148ff9640 1 -- 172.21.15.135:0/1755690240 <== mon.0 v1:172.21.15.135:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f5144003dc0 con 0x7f514c0a9930 2024-03-29T11:34:49.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.908+0000 7f5148ff9640 1 -- 172.21.15.135:0/1755690240 <== mon.0 v1:172.21.15.135:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1288947923 0 0) 0x7f5144084e60 con 0x7f514c0a9930 2024-03-29T11:34:49.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.908+0000 7f5148ff9640 10 client.4578.objecter ms_dispatch 0x7f514c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:34:49.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.908+0000 7f5148ff9640 3 client.4578.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:34:49.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.908+0000 7f5148ff9640 3 client.4578.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:34:49.913 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.908+0000 7f5148ff9640 20 client.4578.objecter dump_active .. 0 homeless 2024-03-29T11:34:49.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.912+0000 7f514b7fe640 10 monclient: get_auth_request con 0x7f512c06db40 auth_method 0 2024-03-29T11:34:49.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.912+0000 7f5148ff9640 10 client.4578.objecter ms_handle_connect 0x7f512c06db40 2024-03-29T11:34:49.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.915+0000 7f5148ff9640 1 -- 172.21.15.135:0/1755690240 <== mon.0 v1:172.21.15.135:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5144050570 con 0x7f514c0a9930 2024-03-29T11:34:49.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.915+0000 7f5148ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:49.914 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:49.915+0000 7f5148ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:50.238 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:50.241+0000 7f51523c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:50.239 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:50.241+0000 7f51523c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:50.239 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:50.241+0000 7f51523c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.135:6789/0 2024-03-29T11:34:50.239 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:50.241+0000 7f51523c6640 1 -- 172.21.15.135:0/1755690240 --> v1:172.21.15.135:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f514c130f50 con 0x7f514c0a9930 2024-03-29T11:34:50.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:50.243+0000 7f5148ff9640 1 -- 172.21.15.135:0/1755690240 <== mon.0 v1:172.21.15.135:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2351702130) 0x7f5144052670 con 0x7f514c0a9930 2024-03-29T11:34:50.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:50.243+0000 7f5148ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:50.240 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:50.243+0000 7f5148ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:50.240 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:34:50.242 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":11,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:34:26.903077+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:26.903077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:50.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:50.250+0000 7f51523c6640 10 librados: watch_flush enter 2024-03-29T11:34:50.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:50.250+0000 7f51523c6640 10 librados: watch_flush exit 2024-03-29T11:34:50.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:50.250+0000 7f51523c6640 20 client.4578.objecter shutdown clearing up homeless session... 2024-03-29T11:34:50.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:50.250+0000 7f51523c6640 10 client.4578.objecter successfully canceled tick 2024-03-29T11:34:50.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:50.250+0000 7f51523c6640 1 -- 172.21.15.135:0/1755690240 >> v1:172.21.15.135:6816/3469875285 conn(0x7f512c06db40 legacy=0x7f512c070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:50.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:50.250+0000 7f51523c6640 10 monclient: shutdown 2024-03-29T11:34:50.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:50.250+0000 7f51523c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:50.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:50.250+0000 7f51523c6640 1 -- 172.21.15.135:0/1755690240 >> v1:172.21.15.135:6789/0 conn(0x7f514c0a9930 legacy=0x7f514c12eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:50.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:50.250+0000 7f51523c6640 1 -- 172.21.15.135:0/1755690240 shutdown_connections 2024-03-29T11:34:50.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:50.250+0000 7f51523c6640 1 -- 172.21.15.135:0/1755690240 >> 172.21.15.135:0/1755690240 conn(0x7f514c0b35e0 msgr2=0x7f514c14a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:50.249 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:50.251+0000 7f51523c6640 1 -- 172.21.15.135:0/1755690240 shutdown_connections 2024-03-29T11:34:50.249 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:50.251+0000 7f51523c6640 1 -- 172.21.15.135:0/1755690240 wait complete. 2024-03-29T11:34:50.249 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:50.251+0000 7f51523c6640 1 librados: shutdown 2024-03-29T11:34:50.267 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:34:57.074 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:34:57.232 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.234+0000 7f3cad89e640 10 monclient: get_monmap_and_config 2024-03-29T11:34:57.232 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.234+0000 7f3cad89e640 10 monclient: build_initial_monmap 2024-03-29T11:34:57.232 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.234+0000 7f3cad89e640 10 monclient: monmap: 2024-03-29T11:34:57.232 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:34:57.232 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:57.232 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:34:57.234801+0000 2024-03-29T11:34:57.232 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:34:57.234801+0000 2024-03-29T11:34:57.233 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:57.233 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:57.233 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:34:57.233 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:34:57.233 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:34:57.233 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:57.233 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.235+0000 7f3cad89e640 1 Processor -- start 2024-03-29T11:34:57.234 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.235+0000 7f3cad89e640 1 -- start start 2024-03-29T11:34:57.234 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.235+0000 7f3cad89e640 10 monclient: init 2024-03-29T11:34:57.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.236+0000 7f3cad89e640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:57.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.236+0000 7f3cad89e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:34:57.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.236+0000 7f3cad89e640 10 monclient(hunting): picked mon.noname-c con 0x7f3ca80a9750 addr v1:172.21.15.194:6789/0 2024-03-29T11:34:57.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.236+0000 7f3cad89e640 10 monclient(hunting): picked mon.noname-a con 0x7f3ca80adca0 addr v1:172.21.15.135:6789/0 2024-03-29T11:34:57.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.236+0000 7f3cad89e640 10 monclient(hunting): picked mon.noname-b con 0x7f3ca8146a80 addr v1:172.21.15.135:6790/0 2024-03-29T11:34:57.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.238+0000 7f3cad89e640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca80abdc0 con 0x7f3ca80adca0 2024-03-29T11:34:57.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.238+0000 7f3cad89e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca80b0070 con 0x7f3ca80a9750 2024-03-29T11:34:57.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.238+0000 7f3cad89e640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca80af880 con 0x7f3ca8146a80 2024-03-29T11:34:57.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.238+0000 7f3cad89e640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:57.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.238+0000 7f3cad89e640 10 monclient(hunting): authenticate will time out at 1404.606300s 2024-03-29T11:34:57.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.238+0000 7f3cac89c640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f3ca80a9750 0x7f3ca8143710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:47470/0 (socket says 172.21.15.135:47470) 2024-03-29T11:34:57.236 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.238+0000 7f3cac89c640 1 -- 172.21.15.135:0/2627225217 learned_addr learned my addr 172.21.15.135:0/2627225217 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:34:57.244 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.240+0000 7f3ca77fe640 1 -- 172.21.15.135:0/2627225217 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2292558945 0 0) 0x7f3ca80b0070 con 0x7f3ca80a9750 2024-03-29T11:34:57.244 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.240+0000 7f3ca77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:34:57.244 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.240+0000 7f3ca77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:57.244 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.240+0000 7f3ca77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:57.244 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.240+0000 7f3ca77fe640 10 monclient(hunting): my global_id is 4600 2024-03-29T11:34:57.244 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.240+0000 7f3ca77fe640 1 -- 172.21.15.135:0/2627225217 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c94003520 con 0x7f3ca80a9750 2024-03-29T11:34:57.245 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.241+0000 7f3ca77fe640 1 -- 172.21.15.135:0/2627225217 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1746388508 0 0) 0x7f3c94003520 con 0x7f3ca80a9750 2024-03-29T11:34:57.245 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.241+0000 7f3ca77fe640 1 -- 172.21.15.135:0/2627225217 >> v1:172.21.15.135:6790/0 conn(0x7f3ca8146a80 legacy=0x7f3ca8151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:57.245 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.241+0000 7f3ca77fe640 1 -- 172.21.15.135:0/2627225217 >> v1:172.21.15.135:6789/0 conn(0x7f3ca80adca0 legacy=0x7f3ca8146370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:34:57.245 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.242+0000 7f3ca77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:34:57.245 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.242+0000 7f3ca77fe640 1 monclient: found mon.noname-c 2024-03-29T11:34:57.245 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.242+0000 7f3ca77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:57.245 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.242+0000 7f3ca77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:34:57.245 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.242+0000 7f3ca77fe640 1 -- 172.21.15.135:0/2627225217 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca80ac030 con 0x7f3ca80a9750 2024-03-29T11:34:57.245 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.242+0000 7f3ca77fe640 10 monclient: _finish_auth 0 2024-03-29T11:34:57.245 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.242+0000 7f3ca77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:34:57.245 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.242+0000 7f3ca77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:57.246 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.242+0000 7f3ca77fe640 1 -- 172.21.15.135:0/2627225217 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f3ca0001690 con 0x7f3ca80a9750 2024-03-29T11:34:57.246 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.242+0000 7f3ca77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:57.246 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.242+0000 7f3ca77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:57.246 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.242+0000 7f3ca77fe640 10 monclient: dump: 2024-03-29T11:34:57.246 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:57.246 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:57.246 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:57.246 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:57.246 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:57.246 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:57.246 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:57.246 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:57.246 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:57.246 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:57.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.242+0000 7f3ca77fe640 1 -- 172.21.15.135:0/2627225217 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ca0001860 con 0x7f3ca80a9750 2024-03-29T11:34:57.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.242+0000 7f3ca77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:57.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.242+0000 7f3ca77fe640 1 -- 172.21.15.135:0/2627225217 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f3ca0001cb0 con 0x7f3ca80a9750 2024-03-29T11:34:57.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.242+0000 7f3ca77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:57.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.242+0000 7f3ca77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:57.247 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.242+0000 7f3ca77fe640 10 monclient: dump: 2024-03-29T11:34:57.247 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:57.247 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:57.247 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:57.247 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:57.247 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:57.247 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:57.247 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:57.247 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:57.247 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:57.248 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:57.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.242+0000 7f3cad89e640 5 monclient: authenticate success, global_id 4600 2024-03-29T11:34:57.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.242+0000 7f3cad89e640 10 monclient: get_monmap_and_config success 2024-03-29T11:34:57.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.242+0000 7f3cad89e640 4 set_mon_vals no callback set 2024-03-29T11:34:57.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.243+0000 7f3cad89e640 10 monclient: shutdown 2024-03-29T11:34:57.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.243+0000 7f3cad89e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:57.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.243+0000 7f3cad89e640 1 -- 172.21.15.135:0/2627225217 >> v1:172.21.15.194:6789/0 conn(0x7f3ca80a9750 legacy=0x7f3ca8143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:57.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.243+0000 7f3cad89e640 1 -- 172.21.15.135:0/2627225217 shutdown_connections 2024-03-29T11:34:57.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.243+0000 7f3cad89e640 1 -- 172.21.15.135:0/2627225217 >> 172.21.15.135:0/2627225217 conn(0x7f3ca80673e0 msgr2=0x7f3ca80b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:57.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.244+0000 7f3cad89e640 1 -- 172.21.15.135:0/2627225217 shutdown_connections 2024-03-29T11:34:57.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.244+0000 7f3cad89e640 1 -- 172.21.15.135:0/2627225217 wait complete. 2024-03-29T11:34:57.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.245+0000 7f3cad89e640 10 monclient: build_initial_monmap 2024-03-29T11:34:57.248 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.245+0000 7f3cad89e640 10 monclient: monmap: 2024-03-29T11:34:57.248 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:34:57.249 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:34:57.249 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:34:57.249 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:34:57.249 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:57.249 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:57.249 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:34:57.249 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:34:57.249 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:34:57.249 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:57.249 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.246+0000 7f3cad89e640 1 librados: starting msgr at 2024-03-29T11:34:57.249 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.246+0000 7f3cad89e640 1 librados: starting objecter 2024-03-29T11:34:57.249 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.246+0000 7f3cad89e640 1 Processor -- start 2024-03-29T11:34:57.249 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.246+0000 7f3cad89e640 1 -- start start 2024-03-29T11:34:57.250 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.246+0000 7f3cad89e640 1 librados: setting wanted keys 2024-03-29T11:34:57.250 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.246+0000 7f3cad89e640 1 librados: calling monclient init 2024-03-29T11:34:57.250 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.246+0000 7f3cad89e640 10 monclient: init 2024-03-29T11:34:57.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.247+0000 7f3cad89e640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:57.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.248+0000 7f3cad89e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:34:57.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.248+0000 7f3cad89e640 10 monclient(hunting): picked mon.noname-b con 0x7f3ca80adca0 addr v1:172.21.15.194:6789/0 2024-03-29T11:34:57.251 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.248+0000 7f3cad89e640 10 monclient(hunting): picked mon.noname-a con 0x7f3ca8146a80 addr v1:172.21.15.135:6789/0 2024-03-29T11:34:57.252 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.248+0000 7f3cad89e640 10 monclient(hunting): picked mon.noname-c con 0x7f3ca813d280 addr v1:172.21.15.135:6790/0 2024-03-29T11:34:57.252 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.248+0000 7f3cad89e640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca81df670 con 0x7f3ca8146a80 2024-03-29T11:34:57.252 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.248+0000 7f3cad89e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca8072590 con 0x7f3ca80adca0 2024-03-29T11:34:57.252 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.248+0000 7f3cad89e640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ca813cfd0 con 0x7f3ca813d280 2024-03-29T11:34:57.252 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.248+0000 7f3cad89e640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:57.252 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.248+0000 7f3cad89e640 10 monclient(hunting): authenticate will time out at 1404.616872s 2024-03-29T11:34:57.252 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.248+0000 7f3cac89c640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f3ca80adca0 0x7f3ca80701a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:57524/0 (socket says 172.21.15.135:57524) 2024-03-29T11:34:57.252 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.252+0000 7f3cac89c640 1 -- 172.21.15.135:0/2274164672 learned_addr learned my addr 172.21.15.135:0/2274164672 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:34:57.252 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.252+0000 7f3ca57fa640 10 client.?.objecter ms_handle_connect 0x7f3ca80adca0 2024-03-29T11:34:57.252 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.252+0000 7f3ca57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:57.257 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.253+0000 7f3ca57fa640 10 client.?.objecter ms_handle_connect 0x7f3ca813d280 2024-03-29T11:34:57.257 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.253+0000 7f3ca57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:57.257 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.253+0000 7f3ca57fa640 1 -- 172.21.15.135:0/2274164672 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3156912880 0 0) 0x7f3ca8072590 con 0x7f3ca80adca0 2024-03-29T11:34:57.257 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.253+0000 7f3ca57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:34:57.257 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.253+0000 7f3ca57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:57.257 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.253+0000 7f3ca57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:57.257 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.253+0000 7f3ca57fa640 10 monclient(hunting): my global_id is 4603 2024-03-29T11:34:57.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.253+0000 7f3ca57fa640 1 -- 172.21.15.135:0/2274164672 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c88004eb0 con 0x7f3ca80adca0 2024-03-29T11:34:57.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 1 -- 172.21.15.135:0/2274164672 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 888180182 0 0) 0x7f3ca813cfd0 con 0x7f3ca813d280 2024-03-29T11:34:57.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:34:57.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:34:57.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:34:57.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:34:57.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 1 -- 172.21.15.135:0/2274164672 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ca8072590 con 0x7f3ca813d280 2024-03-29T11:34:57.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 1 -- 172.21.15.135:0/2274164672 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1274765233 0 0) 0x7f3ca8072590 con 0x7f3ca813d280 2024-03-29T11:34:57.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 1 -- 172.21.15.135:0/2274164672 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c88005490 con 0x7f3ca813d280 2024-03-29T11:34:57.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 1 -- 172.21.15.135:0/2274164672 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f3c90001790 con 0x7f3ca813d280 2024-03-29T11:34:57.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:57.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:34:57.258 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 10 monclient(hunting): dump: 2024-03-29T11:34:57.258 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:57.259 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:57.259 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:57.259 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:57.259 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:57.259 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:57.259 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:57.259 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:57.259 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:57.259 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:57.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:34:57.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 1 -- 172.21.15.135:0/2274164672 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3236991948 0 0) 0x7f3c88004eb0 con 0x7f3ca80adca0 2024-03-29T11:34:57.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 1 -- 172.21.15.135:0/2274164672 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c88005b60 con 0x7f3ca80adca0 2024-03-29T11:34:57.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 1 -- 172.21.15.135:0/2274164672 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f3ca0001b70 con 0x7f3ca80adca0 2024-03-29T11:34:57.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:57.259 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:57.260 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 10 monclient(hunting): dump: 2024-03-29T11:34:57.260 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:57.260 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:57.260 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:57.260 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:57.260 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:57.260 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:57.260 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:57.260 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:57.260 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:57.260 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:57.260 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 10 client.?.objecter ms_handle_connect 0x7f3ca8146a80 2024-03-29T11:34:57.260 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:34:57.260 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 1 -- 172.21.15.135:0/2274164672 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 717880719 0 0) 0x7f3c88005b60 con 0x7f3ca80adca0 2024-03-29T11:34:57.261 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 1 -- 172.21.15.135:0/2274164672 >> v1:172.21.15.135:6790/0 conn(0x7f3ca813d280 legacy=0x7f3ca813c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:57.261 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 1 -- 172.21.15.135:0/2274164672 >> v1:172.21.15.135:6789/0 conn(0x7f3ca8146a80 legacy=0x7f3ca813c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:57.261 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:34:57.261 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 1 monclient: found mon.b 2024-03-29T11:34:57.261 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:34:57.261 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:34:57.261 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 1 -- 172.21.15.135:0/2274164672 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ca813dd20 con 0x7f3ca80adca0 2024-03-29T11:34:57.261 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 10 monclient: _finish_auth 0 2024-03-29T11:34:57.261 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:34:57.261 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3ca57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:34:57.261 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3cad89e640 5 monclient: authenticate success, global_id 4603 2024-03-29T11:34:57.261 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3cad89e640 10 monclient: _renew_subs 2024-03-29T11:34:57.261 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3cad89e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:34:57.261 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.254+0000 7f3cad89e640 1 -- 172.21.15.135:0/2274164672 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ca80708b0 con 0x7f3ca80adca0 2024-03-29T11:34:57.261 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.255+0000 7f3cad89e640 10 client.4603.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:34:57.262 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.255+0000 7f3cad89e640 10 monclient: _renew_subs 2024-03-29T11:34:57.262 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.255+0000 7f3cad89e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:34:57.262 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.255+0000 7f3cad89e640 1 -- 172.21.15.135:0/2274164672 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ca81dee50 con 0x7f3ca80adca0 2024-03-29T11:34:57.262 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.255+0000 7f3cad89e640 1 librados: init done 2024-03-29T11:34:57.262 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.255+0000 7f3ca57fa640 1 -- 172.21.15.135:0/2274164672 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ca0001cf0 con 0x7f3ca80adca0 2024-03-29T11:34:57.262 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.255+0000 7f3ca57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:34:57.262 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.255+0000 7f3ca57fa640 1 -- 172.21.15.135:0/2274164672 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f3ca0004d90 con 0x7f3ca80adca0 2024-03-29T11:34:57.262 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.255+0000 7f3ca57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:34:57.262 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.255+0000 7f3ca57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:34:57.262 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.255+0000 7f3ca57fa640 10 monclient: dump: 2024-03-29T11:34:57.262 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:34:57.262 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:34:57.262 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:57.263 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:34:57.263 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:34:57.263 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:34:57.263 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:34:57.263 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:34:57.263 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:34:57.263 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:34:57.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.255+0000 7f3ca67fc640 4 set_mon_vals no callback set 2024-03-29T11:34:57.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.256+0000 7f3cad89e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:57.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.256+0000 7f3cad89e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:57.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.256+0000 7f3cad89e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:34:57.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.256+0000 7f3cad89e640 1 -- 172.21.15.135:0/2274164672 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c8c005080 con 0x7f3ca80adca0 2024-03-29T11:34:57.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.257+0000 7f3ca57fa640 1 -- 172.21.15.135:0/2274164672 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f3ca001ad60 con 0x7f3ca80adca0 2024-03-29T11:34:57.263 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.258+0000 7f3ca57fa640 1 -- 172.21.15.135:0/2274164672 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1288947923 0 0) 0x7f3ca0084740 con 0x7f3ca80adca0 2024-03-29T11:34:57.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.258+0000 7f3ca57fa640 10 client.4603.objecter ms_dispatch 0x7f3ca8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:34:57.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.258+0000 7f3ca57fa640 3 client.4603.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:34:57.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.258+0000 7f3ca57fa640 3 client.4603.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:34:57.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.258+0000 7f3ca57fa640 20 client.4603.objecter dump_active .. 0 homeless 2024-03-29T11:34:57.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.258+0000 7f3ca7fff640 10 monclient: get_auth_request con 0x7f3c8807f670 auth_method 0 2024-03-29T11:34:57.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.259+0000 7f3ca57fa640 10 client.4603.objecter ms_handle_connect 0x7f3c8807f670 2024-03-29T11:34:57.264 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.265+0000 7f3ca57fa640 1 -- 172.21.15.135:0/2274164672 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3ca0050160 con 0x7f3ca80adca0 2024-03-29T11:34:57.265 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.265+0000 7f3ca57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:34:57.265 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.265+0000 7f3ca57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:34:57.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.588+0000 7f3cad89e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:57.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.588+0000 7f3cad89e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:57.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.588+0000 7f3cad89e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:34:57.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.588+0000 7f3cad89e640 1 -- 172.21.15.135:0/2274164672 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c8c005330 con 0x7f3ca80adca0 2024-03-29T11:34:57.589 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:34:57.589 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":11,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:34:26.903077+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:26.903077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:34:57.590 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.589+0000 7f3ca57fa640 1 -- 172.21.15.135:0/2274164672 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2351702130) 0x7f3ca0050160 con 0x7f3ca80adca0 2024-03-29T11:34:57.590 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.589+0000 7f3ca57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:34:57.590 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.589+0000 7f3ca57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:34:57.590 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.592+0000 7f3cad89e640 10 librados: watch_flush enter 2024-03-29T11:34:57.590 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.592+0000 7f3cad89e640 10 librados: watch_flush exit 2024-03-29T11:34:57.590 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.592+0000 7f3cad89e640 20 client.4603.objecter shutdown clearing up homeless session... 2024-03-29T11:34:57.590 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.592+0000 7f3cad89e640 10 client.4603.objecter successfully canceled tick 2024-03-29T11:34:57.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.592+0000 7f3cad89e640 1 -- 172.21.15.135:0/2274164672 >> v1:172.21.15.135:6816/3469875285 conn(0x7f3c8807f670 legacy=0x7f3c88081b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:57.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.592+0000 7f3cad89e640 10 monclient: shutdown 2024-03-29T11:34:57.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.592+0000 7f3cad89e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:34:57.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.592+0000 7f3cad89e640 1 -- 172.21.15.135:0/2274164672 >> v1:172.21.15.194:6789/0 conn(0x7f3ca80adca0 legacy=0x7f3ca80701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:57.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.592+0000 7f3cad89e640 1 -- 172.21.15.135:0/2274164672 shutdown_connections 2024-03-29T11:34:57.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.592+0000 7f3cad89e640 1 -- 172.21.15.135:0/2274164672 >> 172.21.15.135:0/2274164672 conn(0x7f3ca80673e0 msgr2=0x7f3ca80aa250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:34:57.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.592+0000 7f3cad89e640 1 -- 172.21.15.135:0/2274164672 shutdown_connections 2024-03-29T11:34:57.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.592+0000 7f3cad89e640 1 -- 172.21.15.135:0/2274164672 wait complete. 2024-03-29T11:34:57.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:34:57.593+0000 7f3cad89e640 1 librados: shutdown 2024-03-29T11:34:57.608 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:04.415 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:04.571 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.574+0000 7f4e1292e640 10 monclient: get_monmap_and_config 2024-03-29T11:35:04.571 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.574+0000 7f4e1292e640 10 monclient: build_initial_monmap 2024-03-29T11:35:04.571 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.574+0000 7f4e1292e640 10 monclient: monmap: 2024-03-29T11:35:04.572 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:35:04.572 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:35:04.572 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:35:04.574769+0000 2024-03-29T11:35:04.572 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:35:04.574769+0000 2024-03-29T11:35:04.572 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:04.572 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:35:04.572 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:35:04.572 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:35:04.572 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:35:04.572 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:35:04.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.575+0000 7f4e1292e640 1 Processor -- start 2024-03-29T11:35:04.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.575+0000 7f4e1292e640 1 -- start start 2024-03-29T11:35:04.573 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.575+0000 7f4e1292e640 10 monclient: init 2024-03-29T11:35:04.574 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.576+0000 7f4e1292e640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:04.574 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.576+0000 7f4e1292e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:04.574 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.576+0000 7f4e1292e640 10 monclient(hunting): picked mon.noname-b con 0x7f4e0c0b0770 addr v1:172.21.15.135:6790/0 2024-03-29T11:35:04.574 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.576+0000 7f4e1292e640 10 monclient(hunting): picked mon.noname-a con 0x7f4e0c14a700 addr v1:172.21.15.135:6789/0 2024-03-29T11:35:04.574 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.576+0000 7f4e1292e640 10 monclient(hunting): picked mon.noname-c con 0x7f4e0c0ac5a0 addr v1:172.21.15.194:6789/0 2024-03-29T11:35:04.574 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.576+0000 7f4e1292e640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0c0aba70 con 0x7f4e0c14a700 2024-03-29T11:35:04.574 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.576+0000 7f4e1292e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0c0aecb0 con 0x7f4e0c0ac5a0 2024-03-29T11:35:04.574 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.576+0000 7f4e1292e640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0c0af150 con 0x7f4e0c0b0770 2024-03-29T11:35:04.574 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.576+0000 7f4e1292e640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:04.574 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.576+0000 7f4e1292e640 10 monclient(hunting): authenticate will time out at 1411.944542s 2024-03-29T11:35:04.574 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.576+0000 7f4e10ecb640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f4e0c0ac5a0 0x7f4e0c151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:35832/0 (socket says 172.21.15.135:35832) 2024-03-29T11:35:04.575 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.576+0000 7f4e10ecb640 1 -- 172.21.15.135:0/3188391795 learned_addr learned my addr 172.21.15.135:0/3188391795 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:35:04.576 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.577+0000 7f4e0affd640 1 -- 172.21.15.135:0/3188391795 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1527083632 0 0) 0x7f4e0c0aecb0 con 0x7f4e0c0ac5a0 2024-03-29T11:35:04.576 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.578+0000 7f4e0affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:04.576 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.578+0000 7f4e0affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:04.576 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.578+0000 7f4e0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:04.576 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.578+0000 7f4e0affd640 10 monclient(hunting): my global_id is 4606 2024-03-29T11:35:04.576 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.578+0000 7f4e0affd640 1 -- 172.21.15.135:0/3188391795 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4df4003520 con 0x7f4e0c0ac5a0 2024-03-29T11:35:04.576 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.578+0000 7f4e0affd640 1 -- 172.21.15.135:0/3188391795 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429400358 0 0) 0x7f4e0c0af150 con 0x7f4e0c0b0770 2024-03-29T11:35:04.576 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.578+0000 7f4e0affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:04.576 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.578+0000 7f4e0affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:04.576 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.578+0000 7f4e0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:04.577 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.578+0000 7f4e0affd640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:35:04.577 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.578+0000 7f4e0affd640 1 -- 172.21.15.135:0/3188391795 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e0c0aecb0 con 0x7f4e0c0b0770 2024-03-29T11:35:04.577 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.578+0000 7f4e0affd640 1 -- 172.21.15.135:0/3188391795 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 836048824 0 0) 0x7f4e0c0aba70 con 0x7f4e0c14a700 2024-03-29T11:35:04.577 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.578+0000 7f4e0affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:04.577 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.578+0000 7f4e0affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:04.577 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.578+0000 7f4e0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:04.577 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.578+0000 7f4e0affd640 10 monclient(hunting): my global_id is 4581 2024-03-29T11:35:04.577 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.578+0000 7f4e0affd640 1 -- 172.21.15.135:0/3188391795 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e0c0af150 con 0x7f4e0c14a700 2024-03-29T11:35:04.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.579+0000 7f4e0affd640 1 -- 172.21.15.135:0/3188391795 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4097297058 0 0) 0x7f4df4003520 con 0x7f4e0c0ac5a0 2024-03-29T11:35:04.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.579+0000 7f4e0affd640 1 -- 172.21.15.135:0/3188391795 >> v1:172.21.15.135:6790/0 conn(0x7f4e0c0b0770 legacy=0x7f4e0c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:04.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.579+0000 7f4e0affd640 1 -- 172.21.15.135:0/3188391795 >> v1:172.21.15.135:6789/0 conn(0x7f4e0c14a700 legacy=0x7f4e0c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:04.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.579+0000 7f4e0affd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:04.578 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.579+0000 7f4e0affd640 1 monclient: found mon.noname-c 2024-03-29T11:35:04.579 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.579+0000 7f4e0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:04.579 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.579+0000 7f4e0affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.194:6789/0 2024-03-29T11:35:04.579 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.579+0000 7f4e0affd640 1 -- 172.21.15.135:0/3188391795 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0c0ac9d0 con 0x7f4e0c0ac5a0 2024-03-29T11:35:04.579 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.579+0000 7f4e0affd640 10 monclient: _finish_auth 0 2024-03-29T11:35:04.579 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.579+0000 7f4e0affd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:04.579 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.579+0000 7f4e0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:04.579 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.579+0000 7f4e0affd640 1 -- 172.21.15.135:0/3188391795 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f4e040016b0 con 0x7f4e0c0ac5a0 2024-03-29T11:35:04.579 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.579+0000 7f4e0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:04.579 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.579+0000 7f4e0affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:04.579 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.579+0000 7f4e0affd640 10 monclient: dump: 2024-03-29T11:35:04.579 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:35:04.579 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:35:04.579 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:04.579 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:04.580 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:04.580 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:35:04.580 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:35:04.580 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:35:04.580 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:35:04.580 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:35:04.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.579+0000 7f4e1292e640 5 monclient: authenticate success, global_id 4606 2024-03-29T11:35:04.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.579+0000 7f4e1292e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:04.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.579+0000 7f4e0affd640 1 -- 172.21.15.135:0/3188391795 <== mon.1 v1:172.21.15.194:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e04001890 con 0x7f4e0c0ac5a0 2024-03-29T11:35:04.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.579+0000 7f4e0affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:04.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.579+0000 7f4e0affd640 1 -- 172.21.15.135:0/3188391795 <== mon.1 v1:172.21.15.194:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f4e04001ca0 con 0x7f4e0c0ac5a0 2024-03-29T11:35:04.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.579+0000 7f4e0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:04.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.579+0000 7f4e0affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:04.580 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.579+0000 7f4e0affd640 10 monclient: dump: 2024-03-29T11:35:04.581 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:35:04.581 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:35:04.581 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:04.581 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:04.581 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:04.581 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:35:04.581 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:35:04.581 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:35:04.581 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:35:04.581 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:35:04.581 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.579+0000 7f4e1292e640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:04.581 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.579+0000 7f4e1292e640 4 set_mon_vals no callback set 2024-03-29T11:35:04.581 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.581+0000 7f4e1292e640 10 monclient: shutdown 2024-03-29T11:35:04.581 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.581+0000 7f4e1292e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:04.581 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.581+0000 7f4e1292e640 1 -- 172.21.15.135:0/3188391795 >> v1:172.21.15.194:6789/0 conn(0x7f4e0c0ac5a0 legacy=0x7f4e0c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:04.582 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.581+0000 7f4e1292e640 1 -- 172.21.15.135:0/3188391795 shutdown_connections 2024-03-29T11:35:04.582 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.581+0000 7f4e1292e640 1 -- 172.21.15.135:0/3188391795 >> 172.21.15.135:0/3188391795 conn(0x7f4e0c06b660 msgr2=0x7f4e0c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:04.582 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.581+0000 7f4e1292e640 1 -- 172.21.15.135:0/3188391795 shutdown_connections 2024-03-29T11:35:04.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.581+0000 7f4e1292e640 1 -- 172.21.15.135:0/3188391795 wait complete. 2024-03-29T11:35:04.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.582+0000 7f4e1292e640 10 monclient: build_initial_monmap 2024-03-29T11:35:04.585 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.582+0000 7f4e1292e640 10 monclient: monmap: 2024-03-29T11:35:04.585 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:35:04.585 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:04.586 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:35:04.586 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:35:04.586 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:04.586 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:35:04.586 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:35:04.586 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:35:04.586 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:35:04.586 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:35:04.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.582+0000 7f4e1292e640 1 librados: starting msgr at 2024-03-29T11:35:04.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.582+0000 7f4e1292e640 1 librados: starting objecter 2024-03-29T11:35:04.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.583+0000 7f4e1292e640 1 Processor -- start 2024-03-29T11:35:04.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.583+0000 7f4e1292e640 1 -- start start 2024-03-29T11:35:04.586 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.583+0000 7f4e1292e640 1 librados: setting wanted keys 2024-03-29T11:35:04.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.583+0000 7f4e1292e640 1 librados: calling monclient init 2024-03-29T11:35:04.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.583+0000 7f4e1292e640 10 monclient: init 2024-03-29T11:35:04.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.584+0000 7f4e1292e640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:04.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.584+0000 7f4e1292e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:04.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.584+0000 7f4e1292e640 10 monclient(hunting): picked mon.noname-b con 0x7f4e0c0ac5a0 addr v1:172.21.15.194:6789/0 2024-03-29T11:35:04.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.584+0000 7f4e1292e640 10 monclient(hunting): picked mon.noname-c con 0x7f4e0c0b0770 addr v1:172.21.15.135:6790/0 2024-03-29T11:35:04.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.584+0000 7f4e1292e640 10 monclient(hunting): picked mon.noname-a con 0x7f4e0c14a700 addr v1:172.21.15.135:6789/0 2024-03-29T11:35:04.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.584+0000 7f4e1292e640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0c126570 con 0x7f4e0c14a700 2024-03-29T11:35:04.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.584+0000 7f4e1292e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0c1def30 con 0x7f4e0c0ac5a0 2024-03-29T11:35:04.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.584+0000 7f4e1292e640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0c1deb50 con 0x7f4e0c0b0770 2024-03-29T11:35:04.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.584+0000 7f4e1292e640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:04.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.584+0000 7f4e1292e640 10 monclient(hunting): authenticate will time out at 1411.952586s 2024-03-29T11:35:04.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.584+0000 7f4e0bfff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f4e0c0ac5a0 0x7f4e0c126d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.135:36592/0 (socket says 172.21.15.135:36592) 2024-03-29T11:35:04.587 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.584+0000 7f4e0bfff640 1 -- 172.21.15.135:0/2746005438 learned_addr learned my addr 172.21.15.135:0/2746005438 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:35:04.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.584+0000 7f4e08ff9640 10 client.?.objecter ms_handle_connect 0x7f4e0c0ac5a0 2024-03-29T11:35:04.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.584+0000 7f4e08ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:04.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.585+0000 7f4e08ff9640 10 client.?.objecter ms_handle_connect 0x7f4e0c0b0770 2024-03-29T11:35:04.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.585+0000 7f4e08ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:04.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.585+0000 7f4e08ff9640 10 client.?.objecter ms_handle_connect 0x7f4e0c14a700 2024-03-29T11:35:04.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.585+0000 7f4e08ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:04.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.585+0000 7f4e08ff9640 1 -- 172.21.15.135:0/2746005438 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2763333068 0 0) 0x7f4e0c1def30 con 0x7f4e0c0ac5a0 2024-03-29T11:35:04.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.585+0000 7f4e08ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:04.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.585+0000 7f4e08ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:04.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.585+0000 7f4e08ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:04.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.585+0000 7f4e08ff9640 10 monclient(hunting): my global_id is 4609 2024-03-29T11:35:04.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.585+0000 7f4e08ff9640 1 -- 172.21.15.135:0/2746005438 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4df8004eb0 con 0x7f4e0c0ac5a0 2024-03-29T11:35:04.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.586+0000 7f4e08ff9640 1 -- 172.21.15.135:0/2746005438 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3584189040 0 0) 0x7f4e0c126570 con 0x7f4e0c14a700 2024-03-29T11:35:04.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.586+0000 7f4e08ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:04.588 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.586+0000 7f4e08ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:04.589 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.586+0000 7f4e08ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:04.589 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.586+0000 7f4e08ff9640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:35:04.589 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.586+0000 7f4e08ff9640 1 -- 172.21.15.135:0/2746005438 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e0c1def30 con 0x7f4e0c14a700 2024-03-29T11:35:04.589 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.586+0000 7f4e08ff9640 1 -- 172.21.15.135:0/2746005438 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2094984962 0 0) 0x7f4e0c1deb50 con 0x7f4e0c0b0770 2024-03-29T11:35:04.589 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.586+0000 7f4e08ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:04.589 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.586+0000 7f4e08ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:04.589 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.586+0000 7f4e08ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:04.589 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.586+0000 7f4e08ff9640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:35:04.589 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.586+0000 7f4e08ff9640 1 -- 172.21.15.135:0/2746005438 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e0c126570 con 0x7f4e0c0b0770 2024-03-29T11:35:04.589 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.586+0000 7f4e08ff9640 1 -- 172.21.15.135:0/2746005438 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2334449035 0 0) 0x7f4df8004eb0 con 0x7f4e0c0ac5a0 2024-03-29T11:35:04.589 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.586+0000 7f4e08ff9640 1 -- 172.21.15.135:0/2746005438 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e0c1deb50 con 0x7f4e0c0ac5a0 2024-03-29T11:35:04.589 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.586+0000 7f4e08ff9640 1 -- 172.21.15.135:0/2746005438 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f4e000019b0 con 0x7f4e0c0ac5a0 2024-03-29T11:35:04.589 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.586+0000 7f4e08ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:04.589 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.586+0000 7f4e08ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:04.590 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.586+0000 7f4e08ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:04.590 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:35:04.590 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:35:04.590 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:04.590 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:04.590 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:04.590 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:35:04.590 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:35:04.590 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:35:04.590 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:35:04.590 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:35:04.590 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.586+0000 7f4e08ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:04.590 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.587+0000 7f4e08ff9640 1 -- 172.21.15.135:0/2746005438 <== mon.1 v1:172.21.15.194:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3352859889 0 0) 0x7f4e0c1deb50 con 0x7f4e0c0ac5a0 2024-03-29T11:35:04.590 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.587+0000 7f4e08ff9640 1 -- 172.21.15.135:0/2746005438 >> v1:172.21.15.135:6790/0 conn(0x7f4e0c0b0770 legacy=0x7f4e0c1df450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:04.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.587+0000 7f4e08ff9640 1 -- 172.21.15.135:0/2746005438 >> v1:172.21.15.135:6789/0 conn(0x7f4e0c14a700 legacy=0x7f4e0c144f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:04.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.587+0000 7f4e08ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:35:04.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.587+0000 7f4e08ff9640 1 monclient: found mon.b 2024-03-29T11:35:04.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.587+0000 7f4e08ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:04.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.587+0000 7f4e08ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:35:04.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.587+0000 7f4e08ff9640 1 -- 172.21.15.135:0/2746005438 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0c1dece0 con 0x7f4e0c0ac5a0 2024-03-29T11:35:04.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.587+0000 7f4e08ff9640 10 monclient: _finish_auth 0 2024-03-29T11:35:04.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.587+0000 7f4e08ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:35:04.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.587+0000 7f4e08ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:04.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.587+0000 7f4e1292e640 5 monclient: authenticate success, global_id 4609 2024-03-29T11:35:04.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.587+0000 7f4e1292e640 10 monclient: _renew_subs 2024-03-29T11:35:04.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.587+0000 7f4e1292e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:35:04.591 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.587+0000 7f4e1292e640 1 -- 172.21.15.135:0/2746005438 --> v1:172.21.15.194:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e0c145690 con 0x7f4e0c0ac5a0 2024-03-29T11:35:04.592 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.587+0000 7f4e1292e640 10 client.4609.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:04.592 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.587+0000 7f4e1292e640 10 monclient: _renew_subs 2024-03-29T11:35:04.592 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.587+0000 7f4e1292e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:35:04.592 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.587+0000 7f4e1292e640 1 -- 172.21.15.135:0/2746005438 --> v1:172.21.15.194:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e0c125f60 con 0x7f4e0c0ac5a0 2024-03-29T11:35:04.592 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.587+0000 7f4e1292e640 1 librados: init done 2024-03-29T11:35:04.592 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.588+0000 7f4e08ff9640 1 -- 172.21.15.135:0/2746005438 <== mon.1 v1:172.21.15.194:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e00001c70 con 0x7f4e0c0ac5a0 2024-03-29T11:35:04.592 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.588+0000 7f4e08ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:04.592 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.588+0000 7f4e0a7fc640 4 set_mon_vals no callback set 2024-03-29T11:35:04.592 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.588+0000 7f4e08ff9640 1 -- 172.21.15.135:0/2746005438 <== mon.1 v1:172.21.15.194:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f4e00003a60 con 0x7f4e0c0ac5a0 2024-03-29T11:35:04.592 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.588+0000 7f4e08ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:04.592 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.588+0000 7f4e08ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:04.592 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.588+0000 7f4e08ff9640 10 monclient: dump: 2024-03-29T11:35:04.592 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:35:04.592 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:35:04.593 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:04.593 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:04.593 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:04.593 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:35:04.593 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:35:04.593 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:35:04.593 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:35:04.593 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:35:04.594 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.589+0000 7f4e1292e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:04.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.589+0000 7f4e1292e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:04.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.589+0000 7f4e1292e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:35:04.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.590+0000 7f4e1292e640 1 -- 172.21.15.135:0/2746005438 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dec005080 con 0x7f4e0c0ac5a0 2024-03-29T11:35:04.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.590+0000 7f4e08ff9640 1 -- 172.21.15.135:0/2746005438 <== mon.1 v1:172.21.15.194:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f4e00003c40 con 0x7f4e0c0ac5a0 2024-03-29T11:35:04.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.591+0000 7f4e0b7fe640 10 monclient: get_auth_request con 0x7f4df806dd80 auth_method 0 2024-03-29T11:35:04.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.591+0000 7f4e08ff9640 1 -- 172.21.15.135:0/2746005438 <== mon.1 v1:172.21.15.194:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1288947923 0 0) 0x7f4e00054950 con 0x7f4e0c0ac5a0 2024-03-29T11:35:04.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.591+0000 7f4e08ff9640 10 client.4609.objecter ms_dispatch 0x7f4e0c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:35:04.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.591+0000 7f4e08ff9640 3 client.4609.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:35:04.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.591+0000 7f4e08ff9640 3 client.4609.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:35:04.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.591+0000 7f4e08ff9640 20 client.4609.objecter dump_active .. 0 homeless 2024-03-29T11:35:04.595 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.592+0000 7f4e08ff9640 10 client.4609.objecter ms_handle_connect 0x7f4df806dd80 2024-03-29T11:35:04.596 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.599+0000 7f4e08ff9640 1 -- 172.21.15.135:0/2746005438 <== mon.1 v1:172.21.15.194:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4e00050bf0 con 0x7f4e0c0ac5a0 2024-03-29T11:35:04.596 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.599+0000 7f4e08ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:04.596 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.599+0000 7f4e08ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:04.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.970+0000 7f4e1292e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:04.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.970+0000 7f4e1292e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:04.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.970+0000 7f4e1292e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.194:6789/0 2024-03-29T11:35:04.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.970+0000 7f4e1292e640 1 -- 172.21.15.135:0/2746005438 --> v1:172.21.15.194:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4dec005330 con 0x7f4e0c0ac5a0 2024-03-29T11:35:04.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.971+0000 7f4e08ff9640 1 -- 172.21.15.135:0/2746005438 <== mon.1 v1:172.21.15.194:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2351702130) 0x7f4e00050bf0 con 0x7f4e0c0ac5a0 2024-03-29T11:35:04.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.971+0000 7f4e08ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:04.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.971+0000 7f4e08ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:04.968 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:35:04.968 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":11,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:34:26.903077+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:26.903077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:04.972 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.974+0000 7f4e1292e640 10 librados: watch_flush enter 2024-03-29T11:35:04.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.975+0000 7f4e1292e640 10 librados: watch_flush exit 2024-03-29T11:35:04.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.975+0000 7f4e1292e640 20 client.4609.objecter shutdown clearing up homeless session... 2024-03-29T11:35:04.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.975+0000 7f4e1292e640 10 client.4609.objecter successfully canceled tick 2024-03-29T11:35:04.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.975+0000 7f4e1292e640 1 -- 172.21.15.135:0/2746005438 >> v1:172.21.15.135:6816/3469875285 conn(0x7f4df806dd80 legacy=0x7f4df8070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:04.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.975+0000 7f4e1292e640 10 monclient: shutdown 2024-03-29T11:35:04.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.975+0000 7f4e1292e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:04.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.975+0000 7f4e1292e640 1 -- 172.21.15.135:0/2746005438 >> v1:172.21.15.194:6789/0 conn(0x7f4e0c0ac5a0 legacy=0x7f4e0c126d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:04.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.975+0000 7f4e1292e640 1 -- 172.21.15.135:0/2746005438 shutdown_connections 2024-03-29T11:35:04.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.975+0000 7f4e1292e640 1 -- 172.21.15.135:0/2746005438 >> 172.21.15.135:0/2746005438 conn(0x7f4e0c06b660 msgr2=0x7f4e0c0b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:04.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.976+0000 7f4e1292e640 1 -- 172.21.15.135:0/2746005438 shutdown_connections 2024-03-29T11:35:04.974 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.976+0000 7f4e1292e640 1 -- 172.21.15.135:0/2746005438 wait complete. 2024-03-29T11:35:04.974 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:04.976+0000 7f4e1292e640 1 librados: shutdown 2024-03-29T11:35:04.988 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:11.794 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:11.945 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.946+0000 7fc441600640 10 monclient: get_monmap_and_config 2024-03-29T11:35:11.945 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.946+0000 7fc441600640 10 monclient: build_initial_monmap 2024-03-29T11:35:11.945 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.946+0000 7fc441600640 10 monclient: monmap: 2024-03-29T11:35:11.945 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:35:11.945 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:35:11.945 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:35:11.947688+0000 2024-03-29T11:35:11.945 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:35:11.947688+0000 2024-03-29T11:35:11.945 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:11.945 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:35:11.945 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:35:11.946 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:35:11.946 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:35:11.946 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:35:11.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.947+0000 7fc441600640 1 Processor -- start 2024-03-29T11:35:11.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.947+0000 7fc441600640 1 -- start start 2024-03-29T11:35:11.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.947+0000 7fc441600640 10 monclient: init 2024-03-29T11:35:11.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.947+0000 7fc441600640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:11.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.947+0000 7fc441600640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:11.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.947+0000 7fc441600640 10 monclient(hunting): picked mon.noname-a con 0x7fc43c0b4330 addr v1:172.21.15.135:6789/0 2024-03-29T11:35:11.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.947+0000 7fc441600640 10 monclient(hunting): picked mon.noname-b con 0x7fc43c1470b0 addr v1:172.21.15.135:6790/0 2024-03-29T11:35:11.948 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.948+0000 7fc441600640 10 monclient(hunting): picked mon.noname-c con 0x7fc43c1439b0 addr v1:172.21.15.194:6789/0 2024-03-29T11:35:11.949 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.948+0000 7fc441600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc43c06f2a0 con 0x7fc43c0b4330 2024-03-29T11:35:11.949 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.948+0000 7fc441600640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc43c146b90 con 0x7fc43c1439b0 2024-03-29T11:35:11.949 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.948+0000 7fc441600640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc43c0b2ca0 con 0x7fc43c1470b0 2024-03-29T11:35:11.949 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.948+0000 7fc441600640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:11.949 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.948+0000 7fc441600640 10 monclient(hunting): authenticate will time out at 1419.317106s 2024-03-29T11:35:11.949 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.948+0000 7fc43affd640 1 --1- >> v1:172.21.15.135:6789/0 conn(0x7fc43c0b4330 0x7fc43c0b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6789/0 says I am v1:172.21.15.135:35330/0 (socket says 172.21.15.135:35330) 2024-03-29T11:35:11.949 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.948+0000 7fc43affd640 1 -- 172.21.15.135:0/1006768808 learned_addr learned my addr 172.21.15.135:0/1006768808 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:35:11.949 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.948+0000 7fc439ffb640 1 -- 172.21.15.135:0/1006768808 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1270684592 0 0) 0x7fc43c06f2a0 con 0x7fc43c0b4330 2024-03-29T11:35:11.949 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.948+0000 7fc439ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:11.949 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.948+0000 7fc439ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:11.949 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.948+0000 7fc439ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:11.949 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.948+0000 7fc439ffb640 10 monclient(hunting): my global_id is 4587 2024-03-29T11:35:11.950 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.948+0000 7fc439ffb640 1 -- 172.21.15.135:0/1006768808 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc424003520 con 0x7fc43c0b4330 2024-03-29T11:35:11.950 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.948+0000 7fc439ffb640 1 -- 172.21.15.135:0/1006768808 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 584266198 0 0) 0x7fc43c0b2ca0 con 0x7fc43c1470b0 2024-03-29T11:35:11.950 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.948+0000 7fc439ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:11.950 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.948+0000 7fc439ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:11.950 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.948+0000 7fc439ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:11.950 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.948+0000 7fc439ffb640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:35:11.950 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.948+0000 7fc439ffb640 1 -- 172.21.15.135:0/1006768808 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc43c06f2a0 con 0x7fc43c1470b0 2024-03-29T11:35:11.950 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.949+0000 7fc439ffb640 1 -- 172.21.15.135:0/1006768808 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3318372588 0 0) 0x7fc424003520 con 0x7fc43c0b4330 2024-03-29T11:35:11.950 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.949+0000 7fc439ffb640 1 -- 172.21.15.135:0/1006768808 >> v1:172.21.15.135:6790/0 conn(0x7fc43c1470b0 legacy=0x7fc43c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:11.950 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.949+0000 7fc439ffb640 1 -- 172.21.15.135:0/1006768808 >> v1:172.21.15.194:6789/0 conn(0x7fc43c1439b0 legacy=0x7fc43c151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:11.950 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.949+0000 7fc439ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:35:11.950 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.949+0000 7fc439ffb640 1 monclient: found mon.noname-a 2024-03-29T11:35:11.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.949+0000 7fc439ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:11.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.949+0000 7fc439ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.135:6789/0 2024-03-29T11:35:11.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.949+0000 7fc439ffb640 1 -- 172.21.15.135:0/1006768808 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc43c06e0c0 con 0x7fc43c0b4330 2024-03-29T11:35:11.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.949+0000 7fc439ffb640 10 monclient: _finish_auth 0 2024-03-29T11:35:11.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.949+0000 7fc439ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:35:11.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.949+0000 7fc439ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:11.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.949+0000 7fc439ffb640 1 -- 172.21.15.135:0/1006768808 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fc434001a10 con 0x7fc43c0b4330 2024-03-29T11:35:11.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.949+0000 7fc439ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:11.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.949+0000 7fc439ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:11.951 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.949+0000 7fc439ffb640 10 monclient: dump: 2024-03-29T11:35:11.951 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:35:11.951 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:35:11.951 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:11.952 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:11.952 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:11.952 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:35:11.952 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:35:11.952 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:35:11.952 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:35:11.952 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:35:11.952 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.949+0000 7fc441600640 5 monclient: authenticate success, global_id 4587 2024-03-29T11:35:11.952 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.949+0000 7fc441600640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:11.964 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.950+0000 7fc439ffb640 1 -- 172.21.15.135:0/1006768808 <== mon.0 v1:172.21.15.135:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc434001b70 con 0x7fc43c0b4330 2024-03-29T11:35:11.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.950+0000 7fc439ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:11.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.950+0000 7fc439ffb640 1 -- 172.21.15.135:0/1006768808 <== mon.0 v1:172.21.15.135:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fc434001fc0 con 0x7fc43c0b4330 2024-03-29T11:35:11.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.950+0000 7fc439ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:11.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.950+0000 7fc439ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:11.965 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.950+0000 7fc439ffb640 10 monclient: dump: 2024-03-29T11:35:11.965 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:35:11.965 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:35:11.965 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:11.965 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:11.965 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:11.966 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:35:11.966 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:35:11.966 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:35:11.966 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:35:11.966 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:35:11.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.950+0000 7fc441600640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:11.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.950+0000 7fc441600640 4 set_mon_vals no callback set 2024-03-29T11:35:11.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.951+0000 7fc441600640 10 monclient: shutdown 2024-03-29T11:35:11.966 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.951+0000 7fc441600640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:11.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.951+0000 7fc441600640 1 -- 172.21.15.135:0/1006768808 >> v1:172.21.15.135:6789/0 conn(0x7fc43c0b4330 legacy=0x7fc43c0b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:11.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.951+0000 7fc441600640 1 -- 172.21.15.135:0/1006768808 shutdown_connections 2024-03-29T11:35:11.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.951+0000 7fc441600640 1 -- 172.21.15.135:0/1006768808 >> 172.21.15.135:0/1006768808 conn(0x7fc43c066a00 msgr2=0x7fc43c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:11.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.951+0000 7fc441600640 1 -- 172.21.15.135:0/1006768808 shutdown_connections 2024-03-29T11:35:11.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.951+0000 7fc441600640 1 -- 172.21.15.135:0/1006768808 wait complete. 2024-03-29T11:35:11.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.952+0000 7fc441600640 10 monclient: build_initial_monmap 2024-03-29T11:35:11.967 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.952+0000 7fc441600640 10 monclient: monmap: 2024-03-29T11:35:11.967 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:35:11.967 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:11.967 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:35:11.967 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:35:11.968 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:11.968 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:35:11.968 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:35:11.968 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:35:11.968 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:35:11.968 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:35:11.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.953+0000 7fc441600640 1 librados: starting msgr at 2024-03-29T11:35:11.968 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.953+0000 7fc441600640 1 librados: starting objecter 2024-03-29T11:35:11.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.953+0000 7fc441600640 1 Processor -- start 2024-03-29T11:35:11.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.953+0000 7fc441600640 1 -- start start 2024-03-29T11:35:11.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.953+0000 7fc441600640 1 librados: setting wanted keys 2024-03-29T11:35:11.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.953+0000 7fc441600640 1 librados: calling monclient init 2024-03-29T11:35:11.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.953+0000 7fc441600640 10 monclient: init 2024-03-29T11:35:11.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.954+0000 7fc441600640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:11.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.954+0000 7fc441600640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:11.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.954+0000 7fc441600640 10 monclient(hunting): picked mon.noname-c con 0x7fc43c0b4330 addr v1:172.21.15.135:6790/0 2024-03-29T11:35:11.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.954+0000 7fc441600640 10 monclient(hunting): picked mon.noname-b con 0x7fc43c1439b0 addr v1:172.21.15.194:6789/0 2024-03-29T11:35:11.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.954+0000 7fc441600640 10 monclient(hunting): picked mon.noname-a con 0x7fc43c1470b0 addr v1:172.21.15.135:6789/0 2024-03-29T11:35:11.969 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.954+0000 7fc441600640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc43c11fcd0 con 0x7fc43c1470b0 2024-03-29T11:35:11.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.954+0000 7fc43affd640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7fc43c0b4330 0x7fc43c1201f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:42832/0 (socket says 172.21.15.135:42832) 2024-03-29T11:35:11.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.954+0000 7fc43affd640 1 -- 172.21.15.135:0/4222543494 learned_addr learned my addr 172.21.15.135:0/4222543494 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:35:11.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.954+0000 7fc441600640 1 -- 172.21.15.135:0/4222543494 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc43c1415e0 con 0x7fc43c1439b0 2024-03-29T11:35:11.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.954+0000 7fc441600640 1 -- 172.21.15.135:0/4222543494 --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc43c141770 con 0x7fc43c0b4330 2024-03-29T11:35:11.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.954+0000 7fc441600640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:11.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.954+0000 7fc441600640 10 monclient(hunting): authenticate will time out at 1419.323834s 2024-03-29T11:35:11.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.954+0000 7fc4237fe640 10 client.?.objecter ms_handle_connect 0x7fc43c0b4330 2024-03-29T11:35:11.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.954+0000 7fc4237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:11.970 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 10 client.?.objecter ms_handle_connect 0x7fc43c1439b0 2024-03-29T11:35:11.971 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:11.971 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 10 client.?.objecter ms_handle_connect 0x7fc43c1470b0 2024-03-29T11:35:11.971 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:11.971 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 1 -- 172.21.15.135:0/4222543494 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2548435463 0 0) 0x7fc43c1415e0 con 0x7fc43c1439b0 2024-03-29T11:35:11.971 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:11.971 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:11.971 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:11.971 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 10 monclient(hunting): my global_id is 4612 2024-03-29T11:35:11.971 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 1 -- 172.21.15.135:0/4222543494 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc428004eb0 con 0x7fc43c1439b0 2024-03-29T11:35:11.972 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 1 -- 172.21.15.135:0/4222543494 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204304082 0 0) 0x7fc43c141770 con 0x7fc43c0b4330 2024-03-29T11:35:11.972 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:11.972 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:11.972 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:11.972 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:35:11.972 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 1 -- 172.21.15.135:0/4222543494 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc43c1415e0 con 0x7fc43c0b4330 2024-03-29T11:35:11.972 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 1 -- 172.21.15.135:0/4222543494 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027981092 0 0) 0x7fc43c11fcd0 con 0x7fc43c1470b0 2024-03-29T11:35:11.972 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:11.972 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:11.972 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:11.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 10 monclient(hunting): my global_id is 4590 2024-03-29T11:35:11.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 1 -- 172.21.15.135:0/4222543494 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc43c141770 con 0x7fc43c1470b0 2024-03-29T11:35:11.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 1 -- 172.21.15.135:0/4222543494 <== mon.1 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2293983736 0 0) 0x7fc428004eb0 con 0x7fc43c1439b0 2024-03-29T11:35:11.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 1 -- 172.21.15.135:0/4222543494 --> v1:172.21.15.194:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc43c11fcd0 con 0x7fc43c1439b0 2024-03-29T11:35:11.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 1 -- 172.21.15.135:0/4222543494 <== mon.1 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fc4300016b0 con 0x7fc43c1439b0 2024-03-29T11:35:11.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:11.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:11.973 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 10 monclient(hunting): dump: 2024-03-29T11:35:11.973 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:35:11.973 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:35:11.973 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:11.974 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:11.974 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:11.974 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:35:11.974 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:35:11.974 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:35:11.974 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:35:11.974 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:35:11.974 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.955+0000 7fc4237fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:11.974 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.956+0000 7fc4237fe640 1 -- 172.21.15.135:0/4222543494 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3271826243 0 0) 0x7fc43c1415e0 con 0x7fc43c0b4330 2024-03-29T11:35:11.974 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.956+0000 7fc4237fe640 1 -- 172.21.15.135:0/4222543494 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc428004eb0 con 0x7fc43c0b4330 2024-03-29T11:35:11.974 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.956+0000 7fc4237fe640 1 -- 172.21.15.135:0/4222543494 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fc434001f40 con 0x7fc43c0b4330 2024-03-29T11:35:11.975 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.956+0000 7fc4237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:11.975 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.956+0000 7fc4237fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:11.975 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.956+0000 7fc4237fe640 10 monclient(hunting): dump: 2024-03-29T11:35:11.975 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:35:11.975 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:35:11.975 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:11.975 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:11.975 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:11.975 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:35:11.975 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:35:11.975 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:35:11.975 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:35:11.976 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:35:11.976 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.956+0000 7fc4237fe640 1 -- 172.21.15.135:0/4222543494 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2227573509 0 0) 0x7fc43c141770 con 0x7fc43c1470b0 2024-03-29T11:35:11.976 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.956+0000 7fc4237fe640 1 -- 172.21.15.135:0/4222543494 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc43c1415e0 con 0x7fc43c1470b0 2024-03-29T11:35:11.976 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.956+0000 7fc4237fe640 1 -- 172.21.15.135:0/4222543494 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fc42c001730 con 0x7fc43c1470b0 2024-03-29T11:35:11.976 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.956+0000 7fc4237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:11.976 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.956+0000 7fc4237fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:11.976 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.956+0000 7fc4237fe640 10 monclient(hunting): dump: 2024-03-29T11:35:11.976 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:35:11.976 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:35:11.976 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:11.976 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:11.976 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:11.976 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:35:11.977 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:35:11.977 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:35:11.977 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:35:11.977 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:35:11.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.956+0000 7fc4237fe640 1 -- 172.21.15.135:0/4222543494 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4082180350 0 0) 0x7fc428004eb0 con 0x7fc43c0b4330 2024-03-29T11:35:11.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.956+0000 7fc4237fe640 1 -- 172.21.15.135:0/4222543494 >> v1:172.21.15.194:6789/0 conn(0x7fc43c1439b0 legacy=0x7fc43c120900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:11.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.956+0000 7fc4237fe640 1 -- 172.21.15.135:0/4222543494 >> v1:172.21.15.135:6789/0 conn(0x7fc43c1470b0 legacy=0x7fc43c1e6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:11.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.957+0000 7fc4237fe640 10 monclient: _finish_hunting 0 2024-03-29T11:35:11.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.957+0000 7fc4237fe640 1 monclient: found mon.c 2024-03-29T11:35:11.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.957+0000 7fc4237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:11.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.957+0000 7fc4237fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:35:11.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.957+0000 7fc4237fe640 1 -- 172.21.15.135:0/4222543494 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc43c1dc660 con 0x7fc43c0b4330 2024-03-29T11:35:11.977 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.957+0000 7fc4237fe640 10 monclient: _finish_auth 0 2024-03-29T11:35:11.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.957+0000 7fc4237fe640 10 monclient: _check_auth_tickets 2024-03-29T11:35:11.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.957+0000 7fc4237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:11.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.957+0000 7fc441600640 5 monclient: authenticate success, global_id 4631 2024-03-29T11:35:11.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.957+0000 7fc441600640 10 monclient: _renew_subs 2024-03-29T11:35:11.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.957+0000 7fc441600640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:35:11.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.957+0000 7fc441600640 1 -- 172.21.15.135:0/4222543494 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc43c11f7b0 con 0x7fc43c0b4330 2024-03-29T11:35:11.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.957+0000 7fc441600640 10 client.4631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:11.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.957+0000 7fc441600640 10 monclient: _renew_subs 2024-03-29T11:35:11.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.957+0000 7fc441600640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:35:11.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.957+0000 7fc441600640 1 -- 172.21.15.135:0/4222543494 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc43c1dbdd0 con 0x7fc43c0b4330 2024-03-29T11:35:11.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.957+0000 7fc441600640 1 librados: init done 2024-03-29T11:35:11.978 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.958+0000 7fc4237fe640 1 -- 172.21.15.135:0/4222543494 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4340020a0 con 0x7fc43c0b4330 2024-03-29T11:35:11.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.958+0000 7fc4217fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:11.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.958+0000 7fc4237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:11.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.958+0000 7fc4397fa640 4 set_mon_vals no callback set 2024-03-29T11:35:11.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.958+0000 7fc4237fe640 1 -- 172.21.15.135:0/4222543494 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7fc434004fb0 con 0x7fc43c0b4330 2024-03-29T11:35:11.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.959+0000 7fc4217fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:11.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.959+0000 7fc4217fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:35:11.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.959+0000 7fc4217fa640 1 -- 172.21.15.135:0/4222543494 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc43c11fe60 con 0x7fc43c0b4330 2024-03-29T11:35:11.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.960+0000 7fc4237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:11.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.960+0000 7fc4237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:11.979 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.960+0000 7fc4237fe640 10 monclient: dump: 2024-03-29T11:35:11.979 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:35:11.979 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:35:11.979 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:11.980 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:11.980 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:11.980 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:35:11.980 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:35:11.980 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:35:11.980 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:35:11.980 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:35:11.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.961+0000 7fc4237fe640 1 -- 172.21.15.135:0/4222543494 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7fc43401af80 con 0x7fc43c0b4330 2024-03-29T11:35:11.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.962+0000 7fc43a7fc640 10 monclient: get_auth_request con 0x7fc42806e200 auth_method 0 2024-03-29T11:35:11.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.962+0000 7fc4237fe640 1 -- 172.21.15.135:0/4222543494 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1288947923 0 0) 0x7fc434084de0 con 0x7fc43c0b4330 2024-03-29T11:35:11.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.962+0000 7fc4237fe640 10 client.4631.objecter ms_dispatch 0x7fc43c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:35:11.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.962+0000 7fc4237fe640 3 client.4631.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:35:11.980 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.962+0000 7fc4237fe640 3 client.4631.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:35:11.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.962+0000 7fc4237fe640 20 client.4631.objecter dump_active .. 0 homeless 2024-03-29T11:35:11.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.962+0000 7fc4237fe640 10 client.4631.objecter ms_handle_connect 0x7fc42806e200 2024-03-29T11:35:11.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.968+0000 7fc4237fe640 1 -- 172.21.15.135:0/4222543494 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc4340504f0 con 0x7fc43c0b4330 2024-03-29T11:35:11.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.968+0000 7fc4237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:11.981 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:11.968+0000 7fc4237fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:12.346 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:12.347+0000 7fc4217fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:12.346 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:12.347+0000 7fc4217fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:12.346 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:12.347+0000 7fc4217fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:35:12.346 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:12.347+0000 7fc4217fa640 1 -- 172.21.15.135:0/4222543494 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc43c075f10 con 0x7fc43c0b4330 2024-03-29T11:35:12.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:12.348+0000 7fc4237fe640 1 -- 172.21.15.135:0/4222543494 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2351702130) 0x7fc4340525f0 con 0x7fc43c0b4330 2024-03-29T11:35:12.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:12.348+0000 7fc4237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:12.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:12.348+0000 7fc4237fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:12.347 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:35:12.347 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":11,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:34:26.903077+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:26.903077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:12.350 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:12.352+0000 7fc4217fa640 10 librados: watch_flush enter 2024-03-29T11:35:12.350 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:12.352+0000 7fc4217fa640 10 librados: watch_flush exit 2024-03-29T11:35:12.350 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:12.352+0000 7fc4217fa640 20 client.4631.objecter shutdown clearing up homeless session... 2024-03-29T11:35:12.350 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:12.352+0000 7fc4217fa640 10 client.4631.objecter successfully canceled tick 2024-03-29T11:35:12.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:12.352+0000 7fc4217fa640 1 -- 172.21.15.135:0/4222543494 >> v1:172.21.15.135:6816/3469875285 conn(0x7fc42806e200 legacy=0x7fc4280706c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:12.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:12.352+0000 7fc4217fa640 10 monclient: shutdown 2024-03-29T11:35:12.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:12.352+0000 7fc4217fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:12.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:12.352+0000 7fc4217fa640 1 -- 172.21.15.135:0/4222543494 >> v1:172.21.15.135:6790/0 conn(0x7fc43c0b4330 legacy=0x7fc43c1201f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:12.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:12.352+0000 7fc4217fa640 1 -- 172.21.15.135:0/4222543494 shutdown_connections 2024-03-29T11:35:12.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:12.352+0000 7fc4217fa640 1 -- 172.21.15.135:0/4222543494 >> 172.21.15.135:0/4222543494 conn(0x7fc43c066a00 msgr2=0x7fc43c0b4710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:12.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:12.352+0000 7fc4217fa640 1 -- 172.21.15.135:0/4222543494 shutdown_connections 2024-03-29T11:35:12.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:12.352+0000 7fc4217fa640 1 -- 172.21.15.135:0/4222543494 wait complete. 2024-03-29T11:35:12.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:12.352+0000 7fc4217fa640 1 librados: shutdown 2024-03-29T11:35:12.367 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:19.175 DEBUG:teuthology.orchestra.run.smithi135:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:19.325 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.326+0000 7f46b3fff640 10 monclient: get_monmap_and_config 2024-03-29T11:35:19.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.327+0000 7f46b3fff640 10 monclient: build_initial_monmap 2024-03-29T11:35:19.326 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.327+0000 7f46b3fff640 10 monclient: monmap: 2024-03-29T11:35:19.326 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:35:19.326 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:35:19.326 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:35:19.328623+0000 2024-03-29T11:35:19.326 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:35:19.328623+0000 2024-03-29T11:35:19.326 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:19.326 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:35:19.326 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:35:19.326 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c 2024-03-29T11:35:19.326 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-b 2024-03-29T11:35:19.326 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:35:19.327 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.328+0000 7f46b3fff640 1 Processor -- start 2024-03-29T11:35:19.327 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.328+0000 7f46b3fff640 1 -- start start 2024-03-29T11:35:19.327 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.328+0000 7f46b3fff640 10 monclient: init 2024-03-29T11:35:19.327 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.328+0000 7f46b3fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:19.327 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.328+0000 7f46b3fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:19.327 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.328+0000 7f46b3fff640 10 monclient(hunting): picked mon.noname-a con 0x7f46ac0389e0 addr v1:172.21.15.135:6789/0 2024-03-29T11:35:19.327 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.328+0000 7f46b3fff640 10 monclient(hunting): picked mon.noname-b con 0x7f46ac038dc0 addr v1:172.21.15.135:6790/0 2024-03-29T11:35:19.328 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.328+0000 7f46b3fff640 10 monclient(hunting): picked mon.noname-c con 0x7f46ac030660 addr v1:172.21.15.194:6789/0 2024-03-29T11:35:19.328 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.328+0000 7f46b3fff640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46ac0c54c0 con 0x7f46ac0389e0 2024-03-29T11:35:19.328 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.329+0000 7f46b3fff640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46ac036430 con 0x7f46ac030660 2024-03-29T11:35:19.328 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.329+0000 7f46b3fff640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46ac039840 con 0x7f46ac038dc0 2024-03-29T11:35:19.328 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.329+0000 7f46b3fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:19.328 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.329+0000 7f46b3fff640 10 monclient(hunting): authenticate will time out at 1426.698065s 2024-03-29T11:35:19.331 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.331+0000 7f46b27fc640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f46ac038dc0 0x7f46ac0cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:53904/0 (socket says 172.21.15.135:53904) 2024-03-29T11:35:19.331 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.331+0000 7f46b27fc640 1 -- 172.21.15.135:0/3008618241 learned_addr learned my addr 172.21.15.135:0/3008618241 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:35:19.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.335+0000 7f46b1ffb640 1 -- 172.21.15.135:0/3008618241 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 336166578 0 0) 0x7f46ac0c54c0 con 0x7f46ac0389e0 2024-03-29T11:35:19.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.335+0000 7f46b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:19.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.335+0000 7f46b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:19.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.335+0000 7f46b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:19.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.335+0000 7f46b1ffb640 10 monclient(hunting): my global_id is 4593 2024-03-29T11:35:19.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.335+0000 7f46b1ffb640 1 -- 172.21.15.135:0/3008618241 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f469c003520 con 0x7f46ac0389e0 2024-03-29T11:35:19.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.335+0000 7f46b1ffb640 1 -- 172.21.15.135:0/3008618241 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894054824 0 0) 0x7f46ac039840 con 0x7f46ac038dc0 2024-03-29T11:35:19.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.335+0000 7f46b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:19.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.335+0000 7f46b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:19.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.335+0000 7f46b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:19.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.335+0000 7f46b1ffb640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:35:19.335 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.335+0000 7f46b1ffb640 1 -- 172.21.15.135:0/3008618241 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46ac0c54c0 con 0x7f46ac038dc0 2024-03-29T11:35:19.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.335+0000 7f46b1ffb640 1 -- 172.21.15.135:0/3008618241 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1962311619 0 0) 0x7f46ac036430 con 0x7f46ac030660 2024-03-29T11:35:19.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.335+0000 7f46b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:19.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.335+0000 7f46b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:19.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.335+0000 7f46b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:19.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.335+0000 7f46b1ffb640 10 monclient(hunting): my global_id is 4615 2024-03-29T11:35:19.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.335+0000 7f46b1ffb640 1 -- 172.21.15.135:0/3008618241 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46ac039840 con 0x7f46ac030660 2024-03-29T11:35:19.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.336+0000 7f46b1ffb640 1 -- 172.21.15.135:0/3008618241 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3932119371 0 0) 0x7f469c003520 con 0x7f46ac0389e0 2024-03-29T11:35:19.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.336+0000 7f46b1ffb640 1 -- 172.21.15.135:0/3008618241 >> v1:172.21.15.135:6790/0 conn(0x7f46ac038dc0 legacy=0x7f46ac0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:19.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.336+0000 7f46b1ffb640 1 -- 172.21.15.135:0/3008618241 >> v1:172.21.15.194:6789/0 conn(0x7f46ac030660 legacy=0x7f46ac0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:19.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.336+0000 7f46b1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:35:19.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.336+0000 7f46b1ffb640 1 monclient: found mon.noname-a 2024-03-29T11:35:19.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.336+0000 7f46b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:19.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.336+0000 7f46b1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.135:6789/0 2024-03-29T11:35:19.336 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.336+0000 7f46b1ffb640 1 -- 172.21.15.135:0/3008618241 --> v1:172.21.15.135:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ac035690 con 0x7f46ac0389e0 2024-03-29T11:35:19.337 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.336+0000 7f46b1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:35:19.337 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.336+0000 7f46b1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:35:19.337 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.336+0000 7f46b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:19.337 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.336+0000 7f46b3fff640 5 monclient: authenticate success, global_id 4593 2024-03-29T11:35:19.337 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.336+0000 7f46b3fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:19.337 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.336+0000 7f46b1ffb640 1 -- 172.21.15.135:0/3008618241 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f46a8001690 con 0x7f46ac0389e0 2024-03-29T11:35:19.337 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.336+0000 7f46b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:19.337 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.336+0000 7f46b1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:19.337 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.336+0000 7f46b1ffb640 10 monclient: dump: 2024-03-29T11:35:19.337 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:35:19.337 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:35:19.337 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:19.337 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:19.337 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:19.337 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:35:19.337 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:35:19.338 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:35:19.338 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:35:19.338 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:35:19.338 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.336+0000 7f46b3fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:19.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.336+0000 7f46b1ffb640 1 -- 172.21.15.135:0/3008618241 <== mon.0 v1:172.21.15.135:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46a80017f0 con 0x7f46ac0389e0 2024-03-29T11:35:19.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.336+0000 7f46b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:19.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.336+0000 7f46b3fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:19.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.336+0000 7f46b3fff640 4 set_mon_vals no callback set 2024-03-29T11:35:19.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.336+0000 7f46b1ffb640 1 -- 172.21.15.135:0/3008618241 <== mon.0 v1:172.21.15.135:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f46a8001c40 con 0x7f46ac0389e0 2024-03-29T11:35:19.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.336+0000 7f46b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:19.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.336+0000 7f46b1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:19.339 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.336+0000 7f46b1ffb640 10 monclient: dump: 2024-03-29T11:35:19.339 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:35:19.339 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:35:19.339 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:19.339 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:19.340 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:19.340 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:35:19.340 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:35:19.340 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:35:19.340 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:35:19.340 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:35:19.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.337+0000 7f46b3fff640 10 monclient: shutdown 2024-03-29T11:35:19.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.337+0000 7f46b3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:19.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.337+0000 7f46b3fff640 1 -- 172.21.15.135:0/3008618241 >> v1:172.21.15.135:6789/0 conn(0x7f46ac0389e0 legacy=0x7f46ac033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:19.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.338+0000 7f46b3fff640 1 -- 172.21.15.135:0/3008618241 shutdown_connections 2024-03-29T11:35:19.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.338+0000 7f46b3fff640 1 -- 172.21.15.135:0/3008618241 >> 172.21.15.135:0/3008618241 conn(0x7f46ac005c00 msgr2=0x7f46ac006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:19.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.338+0000 7f46b3fff640 1 -- 172.21.15.135:0/3008618241 shutdown_connections 2024-03-29T11:35:19.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.339+0000 7f46b3fff640 1 -- 172.21.15.135:0/3008618241 wait complete. 2024-03-29T11:35:19.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.339+0000 7f46b3fff640 10 monclient: build_initial_monmap 2024-03-29T11:35:19.340 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.339+0000 7f46b3fff640 10 monclient: monmap: 2024-03-29T11:35:19.341 INFO:teuthology.misc.health.smithi135.stderr:epoch 0 2024-03-29T11:35:19.341 INFO:teuthology.misc.health.smithi135.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:19.341 INFO:teuthology.misc.health.smithi135.stderr:last_changed 0.000000 2024-03-29T11:35:19.341 INFO:teuthology.misc.health.smithi135.stderr:created 0.000000 2024-03-29T11:35:19.341 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:19.341 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:35:19.341 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.noname-a 2024-03-29T11:35:19.341 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.noname-b 2024-03-29T11:35:19.341 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.noname-c 2024-03-29T11:35:19.341 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:35:19.341 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.340+0000 7f46b3fff640 1 librados: starting msgr at 2024-03-29T11:35:19.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.340+0000 7f46b3fff640 1 librados: starting objecter 2024-03-29T11:35:19.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.340+0000 7f46b3fff640 1 Processor -- start 2024-03-29T11:35:19.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.340+0000 7f46b3fff640 1 -- start start 2024-03-29T11:35:19.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.340+0000 7f46b3fff640 1 librados: setting wanted keys 2024-03-29T11:35:19.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.340+0000 7f46b3fff640 1 librados: calling monclient init 2024-03-29T11:35:19.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.340+0000 7f46b3fff640 10 monclient: init 2024-03-29T11:35:19.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.341+0000 7f46b3fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:19.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.341+0000 7f46b3fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:19.342 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.341+0000 7f46b3fff640 10 monclient(hunting): picked mon.noname-c con 0x7f46ac030660 addr v1:172.21.15.135:6790/0 2024-03-29T11:35:19.343 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.344+0000 7f46b2ffd640 1 --1- >> v1:172.21.15.135:6790/0 conn(0x7f46ac030660 0x7f46ac0c2550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.135:6790/0 says I am v1:172.21.15.135:55724/0 (socket says 172.21.15.135:55724) 2024-03-29T11:35:19.344 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.344+0000 7f46b3fff640 10 monclient(hunting): picked mon.noname-a con 0x7f46ac0389e0 addr v1:172.21.15.135:6789/0 2024-03-29T11:35:19.344 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.345+0000 7f46b3fff640 10 monclient(hunting): picked mon.noname-b con 0x7f46ac0c6d70 addr v1:172.21.15.194:6789/0 2024-03-29T11:35:19.344 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.345+0000 7f46b3fff640 1 -- --> v1:172.21.15.135:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46ac009c60 con 0x7f46ac0389e0 2024-03-29T11:35:19.344 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.345+0000 7f46b3fff640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46ac009ed0 con 0x7f46ac0c6d70 2024-03-29T11:35:19.344 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.345+0000 7f46b3fff640 1 -- --> v1:172.21.15.135:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46ac0c2c60 con 0x7f46ac030660 2024-03-29T11:35:19.344 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.345+0000 7f46b3fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:19.344 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.345+0000 7f46b3fff640 10 monclient(hunting): authenticate will time out at 1426.714078s 2024-03-29T11:35:19.344 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.345+0000 7f46b2ffd640 1 -- 172.21.15.135:0/2283681081 learned_addr learned my addr 172.21.15.135:0/2283681081 (peer_addr_for_me v1:172.21.15.135:0/0) 2024-03-29T11:35:19.344 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.346+0000 7f469bfff640 10 client.?.objecter ms_handle_connect 0x7f46ac030660 2024-03-29T11:35:19.345 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.346+0000 7f469bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:19.346 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.348+0000 7f469bfff640 10 client.?.objecter ms_handle_connect 0x7f46ac0389e0 2024-03-29T11:35:19.347 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.348+0000 7f469bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:19.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.349+0000 7f469bfff640 1 -- 172.21.15.135:0/2283681081 <== mon.2 v1:172.21.15.135:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512407091 0 0) 0x7f46ac0c2c60 con 0x7f46ac030660 2024-03-29T11:35:19.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.349+0000 7f469bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:19.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.349+0000 7f469bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:19.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.349+0000 7f469bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:19.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.349+0000 7f469bfff640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:35:19.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.349+0000 7f469bfff640 1 -- 172.21.15.135:0/2283681081 --> v1:172.21.15.135:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46a0004eb0 con 0x7f46ac030660 2024-03-29T11:35:19.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.349+0000 7f469bfff640 1 -- 172.21.15.135:0/2283681081 <== mon.0 v1:172.21.15.135:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444521211 0 0) 0x7f46ac009c60 con 0x7f46ac0389e0 2024-03-29T11:35:19.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.349+0000 7f469bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:19.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.349+0000 7f469bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:19.348 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.349+0000 7f469bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:19.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.349+0000 7f469bfff640 10 monclient(hunting): my global_id is 4596 2024-03-29T11:35:19.349 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.349+0000 7f469bfff640 1 -- 172.21.15.135:0/2283681081 --> v1:172.21.15.135:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46ac0c2c60 con 0x7f46ac0389e0 2024-03-29T11:35:19.350 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.350+0000 7f469bfff640 10 client.?.objecter ms_handle_connect 0x7f46ac0c6d70 2024-03-29T11:35:19.350 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.350+0000 7f469bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:19.350 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.350+0000 7f469bfff640 1 -- 172.21.15.135:0/2283681081 <== mon.2 v1:172.21.15.135:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2532153053 0 0) 0x7f46a0004eb0 con 0x7f46ac030660 2024-03-29T11:35:19.350 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.350+0000 7f469bfff640 1 -- 172.21.15.135:0/2283681081 --> v1:172.21.15.135:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46ac009c60 con 0x7f46ac030660 2024-03-29T11:35:19.350 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.350+0000 7f469bfff640 1 -- 172.21.15.135:0/2283681081 <== mon.0 v1:172.21.15.135:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 9239415 0 0) 0x7f46ac0c2c60 con 0x7f46ac0389e0 2024-03-29T11:35:19.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.350+0000 7f469bfff640 1 -- 172.21.15.135:0/2283681081 --> v1:172.21.15.135:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46a0004eb0 con 0x7f46ac0389e0 2024-03-29T11:35:19.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.350+0000 7f469bfff640 1 -- 172.21.15.135:0/2283681081 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 26048534 0 0) 0x7f46ac009ed0 con 0x7f46ac0c6d70 2024-03-29T11:35:19.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.350+0000 7f469bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:19.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.350+0000 7f469bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:19.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.350+0000 7f469bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:19.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.350+0000 7f469bfff640 10 monclient(hunting): my global_id is 4618 2024-03-29T11:35:19.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.350+0000 7f469bfff640 1 -- 172.21.15.135:0/2283681081 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46ac0c2c60 con 0x7f46ac0c6d70 2024-03-29T11:35:19.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.350+0000 7f469bfff640 1 -- 172.21.15.135:0/2283681081 <== mon.2 v1:172.21.15.135:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f46a8001c00 con 0x7f46ac030660 2024-03-29T11:35:19.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.350+0000 7f469bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:19.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.350+0000 7f469bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:19.351 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.350+0000 7f469bfff640 10 monclient(hunting): dump: 2024-03-29T11:35:19.351 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:35:19.351 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:35:19.351 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:19.351 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:19.352 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:19.352 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:35:19.352 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:35:19.352 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:35:19.352 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:35:19.352 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:35:19.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.350+0000 7f469bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:19.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.350+0000 7f469bfff640 1 -- 172.21.15.135:0/2283681081 <== mon.0 v1:172.21.15.135:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f46b404fdd0 con 0x7f46ac0389e0 2024-03-29T11:35:19.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.350+0000 7f469bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:19.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.350+0000 7f469bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:19.352 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.350+0000 7f469bfff640 10 monclient(hunting): dump: 2024-03-29T11:35:19.352 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:35:19.352 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:35:19.352 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:19.352 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:19.353 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:19.353 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:35:19.353 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:35:19.353 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:35:19.353 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:35:19.353 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:35:19.353 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f469bfff640 1 -- 172.21.15.135:0/2283681081 <== mon.2 v1:172.21.15.135:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3696584721 0 0) 0x7f46ac009c60 con 0x7f46ac030660 2024-03-29T11:35:19.353 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f469bfff640 1 -- 172.21.15.135:0/2283681081 >> v1:172.21.15.194:6789/0 conn(0x7f46ac0c6d70 legacy=0x7f46ac163bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:19.353 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f469bfff640 1 -- 172.21.15.135:0/2283681081 >> v1:172.21.15.135:6789/0 conn(0x7f46ac0389e0 legacy=0x7f46ac158290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:19.353 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f469bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:35:19.353 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f469bfff640 1 monclient: found mon.c 2024-03-29T11:35:19.353 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f469bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:19.353 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f469bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:35:19.353 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f469bfff640 1 -- 172.21.15.135:0/2283681081 --> v1:172.21.15.135:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ac0c6a50 con 0x7f46ac030660 2024-03-29T11:35:19.353 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f469bfff640 10 monclient: _finish_auth 0 2024-03-29T11:35:19.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f469bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:35:19.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f469bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:19.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f469bfff640 1 -- 172.21.15.135:0/2283681081 <== mon.2 v1:172.21.15.135:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46a8003940 con 0x7f46ac030660 2024-03-29T11:35:19.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f469bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:19.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f46b17fa640 4 set_mon_vals no callback set 2024-03-29T11:35:19.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f469bfff640 1 -- 172.21.15.135:0/2283681081 <== mon.2 v1:172.21.15.135:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 603803820 0 0) 0x7f46a8003d50 con 0x7f46ac030660 2024-03-29T11:35:19.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f469bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:19.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f469bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:19.354 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f469bfff640 10 monclient: dump: 2024-03-29T11:35:19.354 INFO:teuthology.misc.health.smithi135.stderr:epoch 1 2024-03-29T11:35:19.354 INFO:teuthology.misc.health.smithi135.stderr:fsid 8b499097-26c8-45be-955e-1a34503f2b8b 2024-03-29T11:35:19.354 INFO:teuthology.misc.health.smithi135.stderr:last_changed 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:19.354 INFO:teuthology.misc.health.smithi135.stderr:created 2024-03-29T11:23:43.910024+0000 2024-03-29T11:35:19.354 INFO:teuthology.misc.health.smithi135.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:19.354 INFO:teuthology.misc.health.smithi135.stderr:election_strategy: 1 2024-03-29T11:35:19.355 INFO:teuthology.misc.health.smithi135.stderr:0: v1:172.21.15.135:6789/0 mon.a 2024-03-29T11:35:19.355 INFO:teuthology.misc.health.smithi135.stderr:1: v1:172.21.15.194:6789/0 mon.b 2024-03-29T11:35:19.355 INFO:teuthology.misc.health.smithi135.stderr:2: v1:172.21.15.135:6790/0 mon.c 2024-03-29T11:35:19.355 INFO:teuthology.misc.health.smithi135.stderr: 2024-03-29T11:35:19.355 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f46b3fff640 5 monclient: authenticate success, global_id 4637 2024-03-29T11:35:19.355 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f46b3fff640 10 monclient: _renew_subs 2024-03-29T11:35:19.355 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f46b3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:35:19.355 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f46b3fff640 1 -- 172.21.15.135:0/2283681081 --> v1:172.21.15.135:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46ac02ee60 con 0x7f46ac030660 2024-03-29T11:35:19.355 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f46b3fff640 10 client.4637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:19.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f46b3fff640 10 monclient: _renew_subs 2024-03-29T11:35:19.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f46b3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:35:19.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f46b3fff640 1 -- 172.21.15.135:0/2283681081 --> v1:172.21.15.135:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46ac1589a0 con 0x7f46ac030660 2024-03-29T11:35:19.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.351+0000 7f46b3fff640 1 librados: init done 2024-03-29T11:35:19.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.353+0000 7f46b3fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:19.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.353+0000 7f46b3fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:19.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.353+0000 7f46b3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:35:19.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.353+0000 7f46b3fff640 1 -- 172.21.15.135:0/2283681081 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46ac007b90 con 0x7f46ac030660 2024-03-29T11:35:19.357 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.354+0000 7f469bfff640 1 -- 172.21.15.135:0/2283681081 <== mon.2 v1:172.21.15.135:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 689328796 0 0) 0x7f46a8003eb0 con 0x7f46ac030660 2024-03-29T11:35:19.358 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.356+0000 7f469bfff640 1 -- 172.21.15.135:0/2283681081 <== mon.2 v1:172.21.15.135:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1288947923 0 0) 0x7f46a8084bc0 con 0x7f46ac030660 2024-03-29T11:35:19.358 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.356+0000 7f469bfff640 10 client.4637.objecter ms_dispatch 0x7f46b4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:35:19.358 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.356+0000 7f469bfff640 3 client.4637.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:35:19.358 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.356+0000 7f469bfff640 3 client.4637.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:35:19.358 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.356+0000 7f469bfff640 20 client.4637.objecter dump_active .. 0 homeless 2024-03-29T11:35:19.358 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.356+0000 7f46b27fc640 10 monclient: get_auth_request con 0x7f46a006ddc0 auth_method 0 2024-03-29T11:35:19.358 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.357+0000 7f469bfff640 10 client.4637.objecter ms_handle_connect 0x7f46a006ddc0 2024-03-29T11:35:19.363 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.364+0000 7f469bfff640 1 -- 172.21.15.135:0/2283681081 <== mon.2 v1:172.21.15.135:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f46a80504e0 con 0x7f46ac030660 2024-03-29T11:35:19.363 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.364+0000 7f469bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:19.363 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.364+0000 7f469bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:19.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.694+0000 7f46b3fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:19.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.694+0000 7f46b3fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:19.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.694+0000 7f46b3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.135:6790/0 2024-03-29T11:35:19.693 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.694+0000 7f46b3fff640 1 -- 172.21.15.135:0/2283681081 --> v1:172.21.15.135:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46ac0c7cc0 con 0x7f46ac030660 2024-03-29T11:35:19.694 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.696+0000 7f469bfff640 1 -- 172.21.15.135:0/2283681081 <== mon.2 v1:172.21.15.135:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2351702130) 0x7f46ac0c7cc0 con 0x7f46ac030660 2024-03-29T11:35:19.694 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.696+0000 7f469bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:19.695 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.696+0000 7f469bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:19.695 INFO:teuthology.misc.health.smithi135.stdout: 2024-03-29T11:35:19.695 INFO:teuthology.misc.health.smithi135.stdout:{"epoch":11,"fsid":"8b499097-26c8-45be-955e-1a34503f2b8b","created":"2024-03-29T11:24:21.977819+0000","modified":"2024-03-29T11:34:26.903077+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:34:26.903077+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8d8500d-c94b-47b0-82f0-8e2c681dd656","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"bb242707-53a0-44df-88b2-357d7b29d7ff","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"4e79fc1c-c76e-4d76-bba8-5c18621facf1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"335b2809-7389-4a89-b0f2-cfb707d4dd75","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"634e5bb7-ef22-4960-b7e3-0fed0bc2ac99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bcbd29f0-a4d9-4708-b160-5cd17058942f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b27f71ce-28e4-4daf-b25f-6fe58abcb572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e19067d4-da68-4171-8fb9-62f960916334","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:19.699 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.700+0000 7f46b3fff640 10 librados: watch_flush enter 2024-03-29T11:35:19.699 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.700+0000 7f46b3fff640 10 librados: watch_flush exit 2024-03-29T11:35:19.699 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.700+0000 7f46b3fff640 20 client.4637.objecter shutdown clearing up homeless session... 2024-03-29T11:35:19.699 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.700+0000 7f46b3fff640 10 client.4637.objecter successfully canceled tick 2024-03-29T11:35:19.699 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.700+0000 7f46b3fff640 1 -- 172.21.15.135:0/2283681081 >> v1:172.21.15.135:6816/3469875285 conn(0x7f46a006ddc0 legacy=0x7f46a0070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:19.699 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.700+0000 7f46b3fff640 10 monclient: shutdown 2024-03-29T11:35:19.699 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.700+0000 7f46b3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:19.699 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.700+0000 7f46b3fff640 1 -- 172.21.15.135:0/2283681081 >> v1:172.21.15.135:6790/0 conn(0x7f46ac030660 legacy=0x7f46ac0c2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:19.701 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.702+0000 7f46b3fff640 1 -- 172.21.15.135:0/2283681081 shutdown_connections 2024-03-29T11:35:19.701 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.702+0000 7f46b3fff640 1 -- 172.21.15.135:0/2283681081 >> 172.21.15.135:0/2283681081 conn(0x7f46ac005c00 msgr2=0x7f46ac0323b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:19.702 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.703+0000 7f46b3fff640 1 -- 172.21.15.135:0/2283681081 shutdown_connections 2024-03-29T11:35:19.702 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.703+0000 7f46b3fff640 1 -- 172.21.15.135:0/2283681081 wait complete. 2024-03-29T11:35:19.702 INFO:teuthology.misc.health.smithi135.stderr:2024-03-29T11:35:19.703+0000 7f46b3fff640 1 librados: shutdown 2024-03-29T11:35:19.718 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:19.718 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:35:19.720 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:35:19.720 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:35:19.720 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:35:19.755 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:35:19.755 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:35:19.756 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:35:19.829 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:35:19.830 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:35:19.830 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:35:19.900 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:35:19.900 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:35:19.900 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:35:19.971 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:35:19.971 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:35:19.971 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:35:20.017 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:35:20.018 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:35:20.018 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:35:20.048 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:35:20.049 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:35:20.049 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:35:20.124 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:35:20.124 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:35:20.124 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:35:20.192 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:35:20.192 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:35:20.192 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:35:20.192 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:35:20.216 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:35:20.216 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:35:20.216 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:35:20.257 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:35:20.257 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:35:20.257 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:35:20.257 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:35:20.268 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:35:20.268 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:35:20.268 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:35:20.319 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:35:20.319 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:35:20.319 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:35:20.330 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:35:20.330 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:35:20.331 DEBUG:teuthology.orchestra.run.smithi135:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:35:20.396 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi135.front.sepia.ceph.com 2024-03-29T11:35:20.396 DEBUG:teuthology.orchestra.run.smithi135:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:35:24.749 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi135.front.sepia.ceph.com 2024-03-29T11:35:24.749 DEBUG:teuthology.orchestra.run.smithi135:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:35:24.853 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi135.front.sepia.ceph.com 2024-03-29T11:35:24.854 DEBUG:teuthology.orchestra.run.smithi135:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:35:24.927 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi135.front.sepia.ceph.com 2024-03-29T11:35:24.928 DEBUG:teuthology.orchestra.run.smithi135:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:35:24.986 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi194.front.sepia.ceph.com 2024-03-29T11:35:24.986 DEBUG:teuthology.orchestra.run.smithi194:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:35:25.992 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi194.front.sepia.ceph.com 2024-03-29T11:35:25.992 DEBUG:teuthology.orchestra.run.smithi194:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:35:26.070 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi194.front.sepia.ceph.com 2024-03-29T11:35:26.071 DEBUG:teuthology.orchestra.run.smithi194:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:35:26.146 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi194.front.sepia.ceph.com 2024-03-29T11:35:26.147 DEBUG:teuthology.orchestra.run.smithi194:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:35:26.228 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:35:26.230 DEBUG:teuthology.misc:Transferring archived files from smithi135:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622705/data/mon.a.tgz 2024-03-29T11:35:26.230 DEBUG:teuthology.orchestra.run.smithi135:> mktemp 2024-03-29T11:35:26.245 INFO:teuthology.orchestra.run.smithi135.stdout:/tmp/tmp.9UhrQxqBOI 2024-03-29T11:35:26.245 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.9UhrQxqBOI 2024-03-29T11:35:26.364 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 0666 /tmp/tmp.9UhrQxqBOI 2024-03-29T11:35:26.448 DEBUG:teuthology.orchestra.remote:smithi135:/tmp/tmp.9UhrQxqBOI is 211KB 2024-03-29T11:35:26.513 DEBUG:teuthology.orchestra.run.smithi135:> rm -fr /tmp/tmp.9UhrQxqBOI 2024-03-29T11:35:26.527 DEBUG:teuthology.misc:Transferring archived files from smithi135:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622705/data/mon.c.tgz 2024-03-29T11:35:26.527 DEBUG:teuthology.orchestra.run.smithi135:> mktemp 2024-03-29T11:35:26.582 INFO:teuthology.orchestra.run.smithi135.stdout:/tmp/tmp.cQpJZJSnOM 2024-03-29T11:35:26.582 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.cQpJZJSnOM 2024-03-29T11:35:26.681 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 0666 /tmp/tmp.cQpJZJSnOM 2024-03-29T11:35:26.765 DEBUG:teuthology.orchestra.remote:smithi135:/tmp/tmp.cQpJZJSnOM is 211KB 2024-03-29T11:35:26.858 DEBUG:teuthology.orchestra.run.smithi135:> rm -fr /tmp/tmp.cQpJZJSnOM 2024-03-29T11:35:26.872 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622705/data/mon.b.tgz 2024-03-29T11:35:26.872 DEBUG:teuthology.orchestra.run.smithi194:> mktemp 2024-03-29T11:35:26.890 INFO:teuthology.orchestra.run.smithi194.stdout:/tmp/tmp.TMpxotdoWG 2024-03-29T11:35:26.891 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.TMpxotdoWG 2024-03-29T11:35:27.003 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0666 /tmp/tmp.TMpxotdoWG 2024-03-29T11:35:27.095 DEBUG:teuthology.orchestra.remote:smithi194:/tmp/tmp.TMpxotdoWG is 211KB 2024-03-29T11:35:27.189 DEBUG:teuthology.orchestra.run.smithi194:> rm -fr /tmp/tmp.TMpxotdoWG 2024-03-29T11:35:27.204 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:35:27.204 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:35:27.206 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:35:27.285 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:35:27.326 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:35:27.327 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:35:27.329 DEBUG:teuthology.misc:Transferring archived files from smithi135:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622705/remote/smithi135/crash 2024-03-29T11:35:27.330 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:35:27.356 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622705/remote/smithi194/crash 2024-03-29T11:35:27.357 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:35:27.392 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:35:27.393 DEBUG:teuthology.orchestra.run.smithi135:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:35:27.401 DEBUG:teuthology.orchestra.run.smithi194:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:35:27.425 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41888.log 2024-03-29T11:35:27.426 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:35:27.426 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41342.log 2024-03-29T11:35:27.426 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.41888.log: /var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41732.log 2024-03-29T11:35:27.426 INFO:teuthology.orchestra.run.smithi135.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41888.log.gz 2024-03-29T11:35:27.426 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.41342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38800.log 2024-03-29T11:35:27.461 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.41732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40127.log 2024-03-29T11:35:27.461 INFO:teuthology.orchestra.run.smithi135.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41342.log.gz 2024-03-29T11:35:27.461 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.38800.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41732.log.gzgzip 2024-03-29T11:35:27.461 INFO:teuthology.orchestra.run.smithi135.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40205.log 2024-03-29T11:35:27.461 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.40127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41849.log 2024-03-29T11:35:27.461 INFO:teuthology.orchestra.run.smithi135.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38800.log.gz 2024-03-29T11:35:27.462 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.40205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38839.log 2024-03-29T11:35:27.462 INFO:teuthology.orchestra.run.smithi135.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40127.log.gz 2024-03-29T11:35:27.462 INFO:teuthology.orchestra.run.smithi135.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40205.log.gz 2024-03-29T11:35:27.462 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.41849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38956.log 2024-03-29T11:35:27.462 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.38839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41107.log 2024-03-29T11:35:27.462 INFO:teuthology.orchestra.run.smithi135.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41849.log.gz 2024-03-29T11:35:27.462 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.38956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39853.log 2024-03-29T11:35:27.462 INFO:teuthology.orchestra.run.smithi135.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38839.log.gz 2024-03-29T11:35:27.462 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.41107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39190.log 2024-03-29T11:35:27.462 INFO:teuthology.orchestra.run.smithi135.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38956.log.gz 2024-03-29T11:35:27.462 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.39853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39619.log 2024-03-29T11:35:27.462 INFO:teuthology.orchestra.run.smithi135.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41107.log.gz 2024-03-29T11:35:27.462 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.39190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39073.log 2024-03-29T11:35:27.463 INFO:teuthology.orchestra.run.smithi135.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39853.log.gz 2024-03-29T11:35:27.463 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.39619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39502.log 2024-03-29T11:35:27.463 INFO:teuthology.orchestra.run.smithi135.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39190.log.gz 2024-03-29T11:35:27.463 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.39073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40283.log 2024-03-29T11:35:27.463 INFO:teuthology.orchestra.run.smithi135.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39619.log.gz 2024-03-29T11:35:27.463 INFO:teuthology.orchestra.run.smithi135.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39073.log.gz 2024-03-29T11:35:27.463 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40873.log 2024-03-29T11:35:27.463 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.39502.log: /var/log/ceph/ceph-client.admin.40283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41537.log 2024-03-29T11:35:27.463 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.40873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41576.log 2024-03-29T11:35:27.463 INFO:teuthology.orchestra.run.smithi135.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39502.log.gz 2024-03-29T11:35:27.463 INFO:teuthology.orchestra.run.smithi135.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40283.log.gz 2024-03-29T11:35:27.463 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.41537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39034.log 2024-03-29T11:35:27.463 INFO:teuthology.orchestra.run.smithi135.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40873.log.gz 2024-03-29T11:35:27.463 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.41576.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41537.log.gz 2024-03-29T11:35:27.464 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40951.log 2024-03-29T11:35:27.464 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.39034.log: gzip 86.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.41576.log.gz --verbose 2024-03-29T11:35:27.464 INFO:teuthology.orchestra.run.smithi135.stderr: -- /var/log/ceph/ceph-client.admin.41146.log 2024-03-29T11:35:27.464 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.40951.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39034.log.gz 2024-03-29T11:35:27.464 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41654.log 2024-03-29T11:35:27.464 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.41146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41029.log 2024-03-29T11:35:27.464 INFO:teuthology.orchestra.run.smithi135.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40951.log.gz 2024-03-29T11:35:27.464 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.41654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40049.log 2024-03-29T11:35:27.464 INFO:teuthology.orchestra.run.smithi135.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41146.log.gz 2024-03-29T11:35:27.464 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.41029.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41654.log.gz 2024-03-29T11:35:27.464 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39424.log 2024-03-29T11:35:27.464 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.40049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38486.log 2024-03-29T11:35:27.464 INFO:teuthology.orchestra.run.smithi135.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41029.log.gz 2024-03-29T11:35:27.464 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.39424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39580.log 2024-03-29T11:35:27.465 INFO:teuthology.orchestra.run.smithi135.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40049.log.gz 2024-03-29T11:35:27.465 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.38486.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39424.log.gz 2024-03-29T11:35:27.465 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:35:27.465 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.39580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41263.log 2024-03-29T11:35:27.465 INFO:teuthology.orchestra.run.smithi135.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38486.log.gz 2024-03-29T11:35:27.465 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose 86.1% -- -- replaced with /var/log/ceph/ceph-client.admin.39580.log.gz /var/log/ceph/ceph-client.admin.39658.log 2024-03-29T11:35:27.465 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-03-29T11:35:27.465 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.41263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40363.log 2024-03-29T11:35:27.465 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.39658.log: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:35:27.465 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38995.log 2024-03-29T11:35:27.465 INFO:teuthology.orchestra.run.smithi135.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41263.log.gz 2024-03-29T11:35:27.465 INFO:teuthology.orchestra.run.smithi135.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39658.log.gz 2024-03-29T11:35:27.465 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.40363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39463.log 2024-03-29T11:35:27.465 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.38995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40010.log 2024-03-29T11:35:27.466 INFO:teuthology.orchestra.run.smithi135.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40363.log.gz 2024-03-29T11:35:27.466 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.39463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34779.log 2024-03-29T11:35:27.466 INFO:teuthology.orchestra.run.smithi135.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38995.log.gz 2024-03-29T11:35:27.466 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.40010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39151.log 2024-03-29T11:35:27.466 INFO:teuthology.orchestra.run.smithi135.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39463.log.gz 2024-03-29T11:35:27.466 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.34779.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40010.log.gz 2024-03-29T11:35:27.466 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40558.log 2024-03-29T11:35:27.466 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.39151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41420.log 2024-03-29T11:35:27.466 INFO:teuthology.orchestra.run.smithi135.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.34779.log.gz 2024-03-29T11:35:27.466 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.40558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40834.log 2024-03-29T11:35:27.466 INFO:teuthology.orchestra.run.smithi135.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39151.log.gz 2024-03-29T11:35:27.466 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.41420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40714.log 2024-03-29T11:35:27.466 INFO:teuthology.orchestra.run.smithi135.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40558.log.gz 2024-03-29T11:35:27.466 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.40834.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:35:27.466 INFO:teuthology.orchestra.run.smithi135.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41420.log.gz 2024-03-29T11:35:27.467 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.40714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40402.log 2024-03-29T11:35:27.467 INFO:teuthology.orchestra.run.smithi135.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40834.log.gz 2024-03-29T11:35:27.467 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41185.log 2024-03-29T11:35:27.467 INFO:teuthology.orchestra.run.smithi135.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40714.log.gz 2024-03-29T11:35:27.467 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.40402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40480.log 2024-03-29T11:35:27.467 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.41185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41068.log 2024-03-29T11:35:27.467 INFO:teuthology.orchestra.run.smithi135.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40402.log.gz 2024-03-29T11:35:27.467 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.40480.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41185.log.gz 2024-03-29T11:35:27.467 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40675.log 2024-03-29T11:35:27.467 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.41068.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40480.log.gz 2024-03-29T11:35:27.467 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38878.log 2024-03-29T11:35:27.467 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41771.log 2024-03-29T11:35:27.467 INFO:teuthology.orchestra.run.smithi135.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41068.log.gz 2024-03-29T11:35:27.467 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.40675.log: /var/log/ceph/ceph-client.admin.38878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40636.log 2024-03-29T11:35:27.468 INFO:teuthology.orchestra.run.smithi135.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40675.log.gz 2024-03-29T11:35:27.468 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.41771.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38878.log.gz 2024-03-29T11:35:27.468 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41381.log 2024-03-29T11:35:27.468 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41693.log 2024-03-29T11:35:27.468 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.40636.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41771.log.gz 2024-03-29T11:35:27.468 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.41381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40441.log 2024-03-29T11:35:27.468 INFO:teuthology.orchestra.run.smithi135.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40636.log.gz 2024-03-29T11:35:27.468 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.41693.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41381.log.gz 2024-03-29T11:35:27.468 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:35:27.468 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.40441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40753.log 2024-03-29T11:35:27.468 INFO:teuthology.orchestra.run.smithi135.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41693.log.gz 2024-03-29T11:35:27.468 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40322.log 2024-03-29T11:35:27.468 INFO:teuthology.orchestra.run.smithi135.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40441.log.gz 2024-03-29T11:35:27.468 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.40753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40088.log 2024-03-29T11:35:27.469 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.40322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38681.log 2024-03-29T11:35:27.469 INFO:teuthology.orchestra.run.smithi135.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40753.log.gz 2024-03-29T11:35:27.469 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.40088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38642.log 2024-03-29T11:35:27.469 INFO:teuthology.orchestra.run.smithi135.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40322.log.gz 2024-03-29T11:35:27.469 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.38681.log: gzip -5 --verbose 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40088.log.gz 2024-03-29T11:35:27.469 INFO:teuthology.orchestra.run.smithi135.stderr: -- /var/log/ceph/ceph-client.admin.39112.log 2024-03-29T11:35:27.469 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.38642.log: gzip -5 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38681.log.gz 2024-03-29T11:35:27.469 INFO:teuthology.orchestra.run.smithi135.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39229.log 2024-03-29T11:35:27.469 INFO:teuthology.orchestra.run.smithi135.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38642.log.gz 2024-03-29T11:35:27.469 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.39112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41302.log 2024-03-29T11:35:27.469 INFO:teuthology.orchestra.run.smithi135.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39112.log.gz 2024-03-29T11:35:27.469 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.39229.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:35:27.469 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.41302.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39229.log.gz 2024-03-29T11:35:27.469 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40597.log 2024-03-29T11:35:27.469 INFO:teuthology.orchestra.run.smithi135.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41302.log.gz 2024-03-29T11:35:27.470 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41810.log 2024-03-29T11:35:27.470 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-mgr.y.log: /var/log/ceph/ceph-client.admin.40597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38525.log 2024-03-29T11:35:27.470 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.41810.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40597.log.gz 2024-03-29T11:35:27.470 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39346.log 2024-03-29T11:35:27.470 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5/var/log/ceph/ceph-client.admin.38525.log: --verbose 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41810.log.gz 2024-03-29T11:35:27.470 INFO:teuthology.orchestra.run.smithi135.stderr: -- /var/log/ceph/ceph-client.admin.41498.log 2024-03-29T11:35:27.470 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.39346.log: gzip -5 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38525.log.gz 2024-03-29T11:35:27.470 INFO:teuthology.orchestra.run.smithi135.stderr: --verbose -- /var/log/ceph/ceph-client.admin.35070.log 2024-03-29T11:35:27.470 INFO:teuthology.orchestra.run.smithi135.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39346.log.gz 2024-03-29T11:35:27.470 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.41498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39931.log 2024-03-29T11:35:27.470 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.35070.log: gzip -5 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41498.log.gz 2024-03-29T11:35:27.470 INFO:teuthology.orchestra.run.smithi135.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40990.log 2024-03-29T11:35:27.470 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.39931.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35070.log.gz 2024-03-29T11:35:27.471 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:35:27.471 INFO:teuthology.orchestra.run.smithi135.stderr:gzip 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39931.log.gz 2024-03-29T11:35:27.471 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.40990.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41224.log 2024-03-29T11:35:27.471 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41615.log 2024-03-29T11:35:27.471 INFO:teuthology.orchestra.run.smithi135.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40990.log.gz/var/log/ceph/ceph-client.admin.41224.log: 2024-03-29T11:35:27.471 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40244.log 2024-03-29T11:35:27.471 INFO:teuthology.orchestra.run.smithi135.stderr: 85.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.41224.log.gz 2024-03-29T11:35:27.471 INFO:teuthology.orchestra.run.smithi135.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40166.log 2024-03-29T11:35:27.471 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.40244.log: /var/log/ceph/ceph-client.admin.41615.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:35:27.471 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.40166.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40244.log.gz 2024-03-29T11:35:27.471 INFO:teuthology.orchestra.run.smithi135.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41615.log.gz 2024-03-29T11:35:27.471 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38720.log 2024-03-29T11:35:27.471 INFO:teuthology.orchestra.run.smithi135.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40166.log.gz 2024-03-29T11:35:27.471 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:35:27.472 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.38720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39970.log 2024-03-29T11:35:27.472 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-osd.2.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38720.log.gz 2024-03-29T11:35:27.472 INFO:teuthology.orchestra.run.smithi135.stderr: 92.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:35:27.472 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41927.log 2024-03-29T11:35:27.472 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39892.log 2024-03-29T11:35:27.472 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.41927.log: /var/log/ceph/ceph-client.admin.39970.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39970.log.gz 2024-03-29T11:35:27.472 INFO:teuthology.orchestra.run.smithi135.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41927.log.gz 2024-03-29T11:35:27.472 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40793.log 2024-03-29T11:35:27.472 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39814.log 2024-03-29T11:35:27.472 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.39892.log: /var/log/ceph/ceph-client.admin.40793.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39892.log.gz 2024-03-29T11:35:27.472 INFO:teuthology.orchestra.run.smithi135.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40793.log.gz 2024-03-29T11:35:27.472 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38761.log 2024-03-29T11:35:27.472 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.39814.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39814.log.gz 2024-03-29T11:35:27.472 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28956.log 2024-03-29T11:35:27.473 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39268.log 2024-03-29T11:35:27.473 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.38761.log: /var/log/ceph/ceph.tmp-client.admin.28956.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28956.log.gz 2024-03-29T11:35:27.473 INFO:teuthology.orchestra.run.smithi135.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38761.log.gz 2024-03-29T11:35:27.473 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39775.log 2024-03-29T11:35:27.473 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40519.log 2024-03-29T11:35:27.473 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.39268.log: /var/log/ceph/ceph-client.admin.39775.log: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.39268.log.gz 2024-03-29T11:35:27.473 INFO:teuthology.orchestra.run.smithi135.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39697.log 2024-03-29T11:35:27.473 INFO:teuthology.orchestra.run.smithi135.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39775.log.gz 2024-03-29T11:35:27.473 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.40519.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40519.log.gz 2024-03-29T11:35:27.473 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39736.log 2024-03-29T11:35:27.473 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39307.log 2024-03-29T11:35:27.473 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.39697.log: /var/log/ceph/ceph-client.admin.39736.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39697.log.gz 2024-03-29T11:35:27.473 INFO:teuthology.orchestra.run.smithi135.stderr: 87.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39385.log 2024-03-29T11:35:27.474 INFO:teuthology.orchestra.run.smithi135.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39736.log.gz 2024-03-29T11:35:27.474 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.39307.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39307.log.gz 2024-03-29T11:35:27.474 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38603.log 2024-03-29T11:35:27.474 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41459.log 2024-03-29T11:35:27.479 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.39385.log: /var/log/ceph/ceph-client.admin.38603.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39385.log.gz 2024-03-29T11:35:27.479 INFO:teuthology.orchestra.run.smithi135.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38603.log.gz 2024-03-29T11:35:27.479 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38917.log 2024-03-29T11:35:27.479 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.41459.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41459.log.gz 2024-03-29T11:35:27.479 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35173.log 2024-03-29T11:35:27.480 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38564.log 2024-03-29T11:35:27.480 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.38917.log: /var/log/ceph/ceph-client.admin.35173.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38917.log.gz 2024-03-29T11:35:27.480 INFO:teuthology.orchestra.run.smithi135.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35173.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40912.log 2024-03-29T11:35:27.480 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-03-29T11:35:27.480 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.38564.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38564.log.gz 2024-03-29T11:35:27.480 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:35:27.480 INFO:teuthology.orchestra.run.smithi135.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39541.log 2024-03-29T11:35:27.480 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.40912.log: /var/log/ceph/ceph-mon.c.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40912.log.gz 2024-03-29T11:35:27.480 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/ceph-client.admin.39541.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39541.log.gz 2024-03-29T11:35:27.484 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34162.log 2024-03-29T11:35:27.484 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:35:27.484 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.34162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34402.log 2024-03-29T11:35:27.484 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34322.log 2024-03-29T11:35:27.484 INFO:teuthology.orchestra.run.smithi194.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34162.log.gz 2024-03-29T11:35:27.484 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.34402.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:35:27.484 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34282.log 2024-03-29T11:35:27.484 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.34322.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34402.log.gz 2024-03-29T11:35:27.484 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:35:27.485 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.34282.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:35:27.485 INFO:teuthology.orchestra.run.smithi194.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34322.log.gz 2024-03-29T11:35:27.485 INFO:teuthology.orchestra.run.smithi194.stderr: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:35:27.485 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:35:27.485 INFO:teuthology.orchestra.run.smithi194.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34282.log.gz 2024-03-29T11:35:27.485 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5/var/log/ceph/ceph-osd.7.log: --verbose -- /var/log/ceph/ceph-client.admin.34202.log 2024-03-29T11:35:27.485 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34362.log 2024-03-29T11:35:27.485 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.34202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34242.log 2024-03-29T11:35:27.485 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.34362.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34202.log.gz 2024-03-29T11:35:27.485 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:35:27.485 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.34242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34442.log 2024-03-29T11:35:27.485 INFO:teuthology.orchestra.run.smithi194.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34362.log.gz 2024-03-29T11:35:27.485 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-mgr.x.log: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.34242.log.gz 2024-03-29T11:35:27.485 INFO:teuthology.orchestra.run.smithi194.stderr: -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:35:27.486 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.34442.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:35:27.486 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34442.log.gz 2024-03-29T11:35:27.486 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-mon.b.log: 87.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:35:27.486 INFO:teuthology.orchestra.run.smithi194.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:35:27.517 INFO:teuthology.orchestra.run.smithi135.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:35:38.608 INFO:teuthology.orchestra.run.smithi194.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:35:38.996 INFO:teuthology.orchestra.run.smithi194.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:35:39.033 INFO:teuthology.orchestra.run.smithi194.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:35:40.448 INFO:teuthology.orchestra.run.smithi135.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:35:40.766 INFO:teuthology.orchestra.run.smithi135.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:35:41.907 INFO:teuthology.orchestra.run.smithi135.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:35:42.928 INFO:teuthology.orchestra.run.smithi194.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:35:43.628 INFO:teuthology.orchestra.run.smithi135.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:36:14.905 INFO:teuthology.orchestra.run.smithi135.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:36:21.516 INFO:teuthology.orchestra.run.smithi135.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:36:21.518 INFO:teuthology.orchestra.run.smithi135.stderr: 2024-03-29T11:36:21.518 INFO:teuthology.orchestra.run.smithi135.stderr:real 0m54.104s 2024-03-29T11:36:21.518 INFO:teuthology.orchestra.run.smithi135.stderr:user 2m32.774s 2024-03-29T11:36:21.518 INFO:teuthology.orchestra.run.smithi135.stderr:sys 0m5.603s 2024-03-29T11:36:36.232 INFO:teuthology.orchestra.run.smithi194.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:36:36.234 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-29T11:36:36.234 INFO:teuthology.orchestra.run.smithi194.stderr:real 1m8.780s 2024-03-29T11:36:36.234 INFO:teuthology.orchestra.run.smithi194.stderr:user 1m54.026s 2024-03-29T11:36:36.234 INFO:teuthology.orchestra.run.smithi194.stderr:sys 0m4.260s 2024-03-29T11:36:36.234 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:36:36.235 DEBUG:teuthology.misc:Transferring archived files from smithi135:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622705/remote/smithi135/log 2024-03-29T11:36:36.236 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:36:58.123 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622705/remote/smithi194/log 2024-03-29T11:36:58.124 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:37:15.946 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:37:16.152 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=24ba3ecbfb004e10b3d413d8b656ed29 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:37:16.155 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:37:16.166 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:37:16.182 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:37:16.182 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:37:16.196 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:37:16.277 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:37:16.288 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:37:16.288 DEBUG:teuthology.orchestra.run.smithi135:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:37:16.291 DEBUG:teuthology.orchestra.run.smithi194:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:37:16.305 INFO:teuthology.orchestra.run.smithi135.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:37:16.308 INFO:teuthology.orchestra.run.smithi194.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:37:16.310 INFO:teuthology.orchestra.run.smithi135.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:37:16.310 INFO:teuthology.orchestra.run.smithi135.stdout:=============================================================================== 2024-03-29T11:37:16.311 INFO:teuthology.orchestra.run.smithi135.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 54 +468us[ +468us] +/- 95ms 2024-03-29T11:37:16.311 INFO:teuthology.orchestra.run.smithi135.stdout:^* hv02.front.sepia.ceph.com 3 6 377 54 -88us[ -95us] +/- 58ms 2024-03-29T11:37:16.311 INFO:teuthology.orchestra.run.smithi135.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 54 -288us[ -288us] +/- 90ms 2024-03-29T11:37:16.311 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:37:16.313 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:37:16.313 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-03-29T11:37:16.313 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 53 +476us[ +472us] +/- 96ms 2024-03-29T11:37:16.313 INFO:teuthology.orchestra.run.smithi194.stdout:^* hv02.front.sepia.ceph.com 3 6 377 52 -76us[ -80us] +/- 58ms 2024-03-29T11:37:16.313 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 54 -287us[ -292us] +/- 90ms 2024-03-29T11:37:16.313 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:37:16.314 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:37:16.326 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:37:16.326 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:37:16.340 DEBUG:teuthology.orchestra.run.smithi135:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:37:16.424 DEBUG:teuthology.orchestra.run.smithi194:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:37:16.496 DEBUG:teuthology.orchestra.run.smithi135:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:37:16.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:37:16.527 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:37:16.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:37:16.560 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:37:16.569 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:37:16.583 INFO:teuthology.task.internal:Duration was 1218.360951 seconds 2024-03-29T11:37:16.584 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:37:16.599 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:37:16.599 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:37:16.602 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:37:16.645 INFO:teuthology.orchestra.run.smithi135.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:37:16.647 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:37:16.957 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:37:16.958 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi135.front.sepia.ceph.com 2024-03-29T11:37:16.958 DEBUG:teuthology.orchestra.run.smithi135:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:37:16.978 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi194.front.sepia.ceph.com 2024-03-29T11:37:16.978 DEBUG:teuthology.orchestra.run.smithi194:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:37:17.001 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:37:17.001 DEBUG:teuthology.orchestra.run.smithi135:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:37:17.020 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:37:17.070 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:37:17.070 DEBUG:teuthology.orchestra.run.smithi135:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:37:17.088 DEBUG:teuthology.orchestra.run.smithi194:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:37:17.210 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:37:17.222 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:37:17.223 DEBUG:teuthology.orchestra.run.smithi135:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:37:17.250 DEBUG:teuthology.orchestra.run.smithi194:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:37:17.277 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:37:17.287 DEBUG:teuthology.orchestra.run.smithi135:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:37:17.293 DEBUG:teuthology.orchestra.run.smithi194:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:37:17.315 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern = core 2024-03-29T11:37:17.343 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = core 2024-03-29T11:37:17.381 DEBUG:teuthology.orchestra.run.smithi135:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:37:17.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:37:17.413 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:37:17.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:37:17.440 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:37:17.451 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:37:17.453 DEBUG:teuthology.misc:Transferring archived files from smithi135:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622705/remote/smithi135 2024-03-29T11:37:17.453 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:37:17.489 DEBUG:teuthology.misc:Transferring archived files from smithi194:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622705/remote/smithi194 2024-03-29T11:37:17.489 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:37:17.524 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:37:17.524 DEBUG:teuthology.orchestra.run.smithi135:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:37:17.527 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:37:17.573 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:37:17.583 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:37:17.583 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:37:17.593 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:37:17.593 DEBUG:teuthology.orchestra.run.smithi135:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:37:17.595 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:37:17.607 INFO:teuthology.orchestra.run.smithi135.stdout: 265365 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:37 /home/ubuntu/cephtest 2024-03-29T11:37:17.628 INFO:teuthology.orchestra.run.smithi194.stdout: 265594 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:37 /home/ubuntu/cephtest 2024-03-29T11:37:17.629 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:37:17.639 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:37:17.662 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:37:17.684 DEBUG:teuthology.nuke:shortname: smithi135 2024-03-29T11:37:17.685 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:37:17.703 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi135.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622705', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:08:33.968797', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB3FPCTUhwsFHInF9Tv4jgUIcbdIN7FEbfb79V0LlTwdLnCP3Ohx3Nkc0l0n1RnUrjovS7OmvmSzaxp2IxMsJco='} 2024-03-29T11:37:17.707 DEBUG:teuthology.nuke:shortname: smithi194 2024-03-29T11:37:17.707 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:37:17.725 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622705', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:08:33.967377', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCGRy6yfe5LtLCQ8rJvM8g9WvDkJwb+f6+7wdIOVHmvUQjJbOpGy6Cvay+gB1EWaXZ0Us87iEr7S2+5OwPUTSLM='} 2024-03-29T11:37:17.755 INFO:teuthology.orchestra.console.smithi135:Power off 2024-03-29T11:37:17.755 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:37:17.770 INFO:teuthology.orchestra.console.smithi194:Power off 2024-03-29T11:37:17.771 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:37:17.779 DEBUG:teuthology.orchestra.console.smithi135:power off output: Chassis Power Control: Down/Off 2024-03-29T11:37:17.780 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:37:17.794 DEBUG:teuthology.orchestra.console.smithi194:power off output: Chassis Power Control: Down/Off 2024-03-29T11:37:17.794 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:37:17.803 DEBUG:teuthology.orchestra.console.smithi135:check power output: Chassis Power is on 2024-03-29T11:37:17.817 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-03-29T11:37:21.804 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:37:21.818 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:37:21.916 DEBUG:teuthology.orchestra.console.smithi135:check power output: Chassis Power is on 2024-03-29T11:37:21.930 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-03-29T11:37:25.918 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:37:25.932 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:37:26.029 DEBUG:teuthology.orchestra.console.smithi135:check power output: Chassis Power is off 2024-03-29T11:37:26.041 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-03-29T11:37:26.131 INFO:teuthology.orchestra.console.smithi135:Power off completed 2024-03-29T11:37:30.046 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:37:30.156 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is off 2024-03-29T11:37:30.257 INFO:teuthology.orchestra.console.smithi194:Power off completed 2024-03-29T11:37:30.361 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} duration: 1218.3609507083893 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=24ba3ecbfb004e10b3d413d8b656ed29 status: fail success: false 2024-03-29T11:37:30.361 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:37:30.435 INFO:teuthology.run:FAIL